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Abstract 

Reliability  analysts  are  often  faced  with  the  challenge  of  characterizing  the  behavior  of 
system  components  based  on  limited  data.  Any  insight  into  which  model  input  data  is 
most  significant  and  how  much  data  is  necessary  to  achieve  desired  accuracy  requirements 
will  improve  the  efficiency  and  cost  effectiveness  of  the  data  collection  and  data 
characterization  processes.  This  thesis  assesses  potential  significant  factors  in  the 
probabilistic  characterization  of  component  failure  and  repair  behavior  with  respect  to  the 
effect  on  system  availability  estimates.  Potential  factors  were  screened  for  significance 
utilizing  fractional  factorial  and  Plackett-Burman  experimental  designs  for  several  system 
models  developed  using  an  AFOTEC  simulation  program  entitled  RAPTOR. 

Two  input  data  characterization  factors  were  found  to  have  a  significant  affect  on 
availability  estimation  accuracy:  the  size  of  the  structure  and  the  number  of  data  points 
used  for  component  failure  and  repair  distributional  fitting.  Estimation  error  was 
minimized  when  the  stmctures  analyzed  were  small  and  many  data  points  (in  this  case,  25) 
were  used  for  the  distributional  fittings.  Assuming  constant  component  failure  rates  and 
using  empirical  repair  distributions  were  found  to  be  equally  effective  component 
characterization  methods  (pertaining  to  model  availability  estimation  error)  compared  to 
using  automated  software  fitting  tools  (or  ‘wizards’).  The  results  of  this  study  also 
indicate  that  there  is  no  apparent  benefit  in  concentrating  on  ‘important’  components  for 
the  highest  fidelity  distributional  fittings. 
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SENSITIVITY  OF  AVAILABILITY  ESTIMATES 
TO  INPUT  DATA  CHARACTERIZATION 


I.  INTRODUCTION 


Overview 

Reliability,  maintainability,  and  availability  (RM&A)  analysis  plays  an  integral  part  in 
the  design  and  production  of  efficient,  cost-effective  systems.  According  to  Kapur  and 
Lamberson, 

“The  reliability  of  a  system  is  the  probability  that,  when  operating  under  stated 
environmental  conditions,  the  system  will  perform  its  intended  function 
adequately  for  a  specified  time.”  [1:1] 

“Maintainability  is  defined  as  the  probability  that  a  failed  system  can  be  made 
operable  in  a  specified  interval  of  downtime.”  [1:225] 

“Availability  is  defined  as  the  probability  that  a  system  is  operating  satisfactorily  at 
any  point  in  time...”  and  “is  a  measure  of  the  ratio  of  the  operating  time  of  the 
system  to  the  operating  time  plus  the  downtime.”  [1:225] 

The  Department  of  Defense  and  the  Air  Force  conduct  numerous  studies  into  the 

rehability  and  maintainability  of  current  and  future  weapons  systems  in  an  effort  to  control 

RM&A  costs  of  fielded  systems  and  to  verify  RM&A  characteristics  of  systems  which  are 

still  in  development.  One  key  Air  Force  agency  which  conducts  such  studies  is 

Headquarters  Air  Force  Operational  Test  and  Evaluation  Center  (HQ  AFOTEC). 

AFOTEC  manages  a  large  portion  of  the  Air  Force’s  weapons  system  operational 

verification  and  validation  testing. 


1 


In  an  effort  to  describe  a  system’s  RM&A  characteristics,  analysts  frequently  represent 
the  system  with  an  analytical  and/or  simulation  model.  Reliability  analysts  will  base  these 
models  on  observed  component  failure  and  repair  data,  historical  performance  of  similar 
systems,  contractor  estimates,  as  well  as  on  certain  traditional  theoretical  assumptions 
whkh  have  been  developed  in  the  field  of  reliability.  In  an  ideal  circumstance,  data  from 
extensive  testing  will  be  available  for  accurate  probabilistic  characterization  of  the  various 
system  components.  However,  due  to  various  constraints  and  limitations,  the  analyst  is 
often  faced  with  the  challenge  of  characterizing  the  behavior  of  system  components  based 
on  limited  data.  In  this  instance,  the  analyst  will  need  to  make  judgments  as  to  how  best 
characterize  the  input  data  to  obtain  acceptable  analytical  results. 


Background 

Systems  are  frequently  broken  down  into  sub-structures  of  components  for  RM&A 
analysis.  Several  categories  of  component  structures  have  been  defined  in  the  field  of 
reliability.  The  more  common  classes  of  structures  include  series,  parallel,  series-parallel, 
and  complex  stmctures.  A  complex  structure  is  one  that  cannot  be  defined  as  series, 
parallel,  or  series-parallel.  The  simplest  example  of  a  series  system  contains  two 
components  as  shown  in  Figure  1. 


Figure  1 .  Simple  Series  System 
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Given  that  p\  and  P2  (ranging  in  value  from  0  to  1.0)  represent  the  reliability  of 


components  1  and  2,  respectively,  and  that  all  components  operate  independently  of  each 
other,  then  the  system  reliability  function,  /2(p) ,  is 

^(P)  =  Pr  Pa  • 

A  two  component  parallel  system  is  shown  in  Figure  2. 


Figure  2.  Simple  Parallel  System 
In  this  case,  the  system  reliability  function  is 

/t(p)  =  l-[(l-/),).(l-P2)]. 

Series-parallel  systems  consist  of  combinations  of  series  and  parallel  components  in  the 
system.  An  example  is  shown  in  Figure  3. 


Figure  3.  Series-Parallel  System 
The  system  reliability  function  for  this  series-parallel  system  is 

/i(P)  =  [l-(l-Pi)(l-P2)][l-(l-P3)-(l-P4)]P5- 
A  typical  complex  structure  can  be  illustrated  by  a  bridge  structure  as  shown  in  Figure  4. 
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Figure  4.  Bridge  Structure 


The  system  reliability  function  for  a  bridge  structure  is 

MP)  =  1-[(1-A  ^P4)•(1-A•/>3•P5)•(1-P2•F5)■(1-P2^P3•P4)]• 

AS  can  clearly  be  seen,  the  complexity  of  the  system  reliability  function  increases 
significantly  as  the  size  and  complexity  of  the  system  structure  increases. 

Several  analytical  methods  exist  for  determining  steady-state  properties  of  systems  of 
components,  including  Markovian  models,  network  theory,  fault  tree  analysis,  path  and 
cut  set  analysis,  Venn  decomposition,  non-homogenous  Poisson  processes  (NHPP),  and 
power  law  processes,  to  name  a  few.  However,  if  the  system  under  study  is  large  and/or 
complicated,  as  is  often  the  case,  analytical  methods  can  become  cumbersome. 
Furthermore,  most  analytical  methods  provide  insight  only  into  the  system’s  steady-state 
properties,  not  it’s  transient  properties.  The  task  is  further  complicated  when  estimating 
system  availability,  since  component  repair  rates  must  be  considered.  In  such  situations 
where  analytical  methods  are  inadequate  or  overly  cumbersome,  simulation  provides  a 
viable  (and  often  times  preferable)  alternative  [2:112]. 

In  developing  a  simulation  model,  analysts  must  collect  component  failure  and  repair 
rate  data  (and/or  use  existing  data)  and  then  characterize  this  data  to  accurately  represent 
the  true  behavior  of  the  components  of  interest.  More  often  than  not,  this  data  collection 
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process  is  time  consuming  and  expensive.  Any  insight  into  which  model  input  data  is  most 
significant  and  how  much  data  is  necessary  to  achieve  desired  accuracy  requirements 
should  improve  the  efficiency  and  cost  effectiveness  of  the  data  collection  and  data 
characterization  processes. 

Research  Objectives 

The  general  purpose  of  this  study  is  to  provide  insight  into  input  data  characterization 
factors  (such  as  volume  of  data  utilized,  data  fitting  methods,  system  size,  type  of  system 
stracture,  and  component  importance)  which  may  affect  the  accuracy  of  simulation  model 
availability  estimates.  If  we  can  identify  the  key  factors  which  have  a  significant  affect  on 
model  accuracy,  the  analyst  can  focus  more  attention  on  modeling  these  significant  factors 
and  less  on  the  insignificant  factors  when  soliciting  and  characterizing  input  data  for  an 
RM&A  model. 

Questions  which  need  to  be  researched  include: 

(1)  How  much  failure  rate  and  repair  rate  data  are  needed  for  each  component  to 
obtain  a  desired  model  accuracy? 

(2)  Which  data  fitting  techniques  for  characterizing  component  failure  and  repair 
probability  distributions  produce  significant  errors  in  model  accuracy,  and  which  do 
not? 

(3)  Do  all  components  need  the  same  fidelity  of  characterization,  or  can  increased 
efficiency  be  realized  by  focusing  on  only  the  ‘important’  components? 

(4)  Are  the  answers  to  the  above  questions  affected  by  system  size,  the  underlying  tme 
component  failure  distributions,  or  other  system  characteristics? 

Although  the  scope  of  this  effort  does  not  allow  for  a  complete  research  of  the  above 

questions,  much  can  be  ascertained  by  conducting  a  controlled  experiment.  This  research 
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is  intended,  using  a  design  of  experiment  approach,  to  help  identify  the  most  critical  pieces 
of  data  needed  to  ensure  representative  simulation  results.  Many  efficiencies  could  be 
achieved  if  analysts  were  provided  general  input  data  characterization  guidelines  based  on 
experimental  results.  Insights  gained  from  this  research  may  assist  in  the  reduction  of 
expensive  live  testing  and  unproductive  data  collection  through  the  efficient  use  of 
simulation  models. 

The  overall  research  objectives  are  to: 

(1)  Identify  potential  factors  which  affect  availability  model  output  accuracy. 

(2)  Screen  these  potential  factors  to  determine  which  have  a  statistically  significant 

effect  (or  interaction  effect)  on  output  accuracy. 

(3)  Assess  the  magnitude  of  the  significant  effects. 

(4)  Provide  basic  insight  to  analysts  to  aid  in  efficient  input  data  characterization  for 

availability  models. 

Scope 

Although  several  model  output  measures  may  be  of  interest  when  analyzing  a  system, 
this  study  focused  on  the  system  availability  output  measure.  A  total  of  nine  input  data 
characterization  factors  (defined  in  Chapter  3),  identified  by  several  RM&A  analysis 
experts  and  the  author  as  factors  with  a  potential  affect  on  the  accuracy  of  availability 
estimates,  were  analyzed.  The  probability  density  functions  (pdf)  used  to  define  system 
component  failure  and  repair  rates  were  limited  to  ‘common’  functions  encountered  in 
reliability  analysis,  namely  the  Weibull  and  Lognormal  pdf’s.  Component  sparing  was  not 
considered  in  this  research.  To  maintain  economy  of  effort,  the  maximum  size  of  any 
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analyzed  system  was  limited  to  20  total  components  and  the  structure  types  analyzed  were 
series-parallel  and  complex. 

Overview  of  Subsequent  Chapters 

Chapter  2  contains  a  review  of  existing  literature  covering  several  topics  pertinent  to 
this  research.  Major  component  importance  measures,  experimental  designs  for  simulation 
(including  screening  designs),  Plackett-Burman  two-level  experimental  screening  designs, 
and  past  research  relating  to  this  effort  are  all  explored. 

The  research  was  conducted  in  two  stages;  a  preliminary  experiment  to  validate  and 
refine  the  methodology,  followed  by  a  larger-scale  experiment.  Chapter  3  includes  a 
description  of  the  research  methodology  for  the  preliminary  experiment  which  assessed 
five  input  data  characterization  factors.  Chapter  3  also  includes  a  discussion  of  the 
specific  designed  experimental  screening  methods  used  as  well  as  specific  analytical 
techniques  used  for  data  analysis  for  the  preliminary  experiment.  The  software  used  for 
availability  model  development,  random  variate  generation,  and  data  fitting  are  described. 

Chapter  4  contains  the  results  from  the  preliminary  experiment.  Statistical  results  are 
presented  which  identify  the  factors  which  proved  significant  in  affecting  availability  model 
output  accuracy. 

Chapters  5  and  6  include  descriptions  of  the  methodology  refinements  and  results  of 
the  final  experiment.  This  experiment  analyzed  nine  input  data  characterization  factors. 


7 


Chapter  7  contains  a  summary  of  the  thesis  effort,  including  an  overview  and 
discussion  of  the  impact  of  the  results,  how  these  results  may  benefit  reliability  analysts, 
and  ideas  for  future  research. 
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11.  LITERATURE  REVIEW 


Overview 

This  chapter  provides  an  overview  of  the  current  literature  in  areas  pertaining  to  this 
thesis.  This  chapter  begins  by  reviewing  several  major  methods  of  defining  component 
importance  which  are  found  in  the  literature.  It  then  provides  an  overview  of  two-level 
designed  experimental  methods  for  factor  screening  in  simulation  experiments.  One 
screening  experimental  technique,  Plackett-Burman  (P-B)  experimental  designs,  was  used 
in  this  research  and  is  discussed  in  detail.  Finally,  past  research  which  relate  to  this  effort 
are  reviewed. 

Component  Importance  Measures 

Systems  are  frequently  broken  down  into  sub-structures  of  components  to  aid  in 
system  design,  analysis,  and  repair.  Component  importance  measures  provide  a  scientific, 
quantitative  approach  of  identifying  the  most  important  components  in  a  given  structure  of 
components.  As  an  example  of  a  common  application,  system  designers  can  use 
component  importance  measure  to  identify  which  components  are  most  eritical  in  the 
proposed  design  structure.  Furthermore,  reliability  analysts  can  use  component  important 
measures  to  determine  which  components  are  most  erucial  in  defining  the  overall  system 
reliability  [3:195]. 

Several  component  importance  measures  have  been  developed  in  reliability  theory  since 
Birnbaum  introduced  the  first  mathematieal  component  importance  measures  in  1969. 
Current  component  importance  measures  can  be  categorized  into  three  areas:  structural, 
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time  dependent,  and  time  independent.  This  section  provides  an  outline  of  several  of  the 
major  component  importance  measures  which  have  been  published  in  recent  years  and  are 
common  in  use. 

Terminology.  All  systems  considered  in  this  paper  are  coherent  systems  comprised  of 
binary  state  components.  A  coherent  system  is  one  in  which  all  components  are  relevant 
in  maintaining  a  functional  system.  Binary  state  components  have  just  two  states: 
functioning  or  failed.  The  states  are  typically  represented  as 

X(0  =  1  if  the  component  functions  at  time  t 
=  0  if  the  component  is  failed  at  time  L 

A  system’s  (as  opposed  to  a  component’s)  reliability  function  is  depicted  as  /z(p),  where  p 
represents  the  component  reliability  vector.  A  component’s  reliability  function  is  a 
function  of  time  and  is  depicted  as  P(i)(t)  for  component  i. 

Structural  Component  Importance  Measures.  Stmctural  importance  measures  are 
based  solely  upon  the  stmctural  design  of  the  system.  They  are  used  when  the  system 
stmcture  function  is  known,  but  the  individual  component  reliabilities  are  not  known 
[4:583].  Two  key  stmctural  methods  have  been  developed  by  Birnbaum  as  well  as 
Barlow  and  Proschan. 

Birnbaum  Structural  Measure.  The  Birnbaum  structural  measure  provides  a 
measure  of  the  criticality  of  a  component  in  maintaining  a  system’s  functional  state. 
Annotated  as  / g  for  component  i,  it  represents  the  proportion  of  system  state  vectors 

which  are  critical  for  component  i  [5:456].  When  the  system  components  are 
independent,  it  can  be  calculated  by  the  following  equation  [4:584]: 
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(1) 


(,.)  _dh(p) 

-  -V 

O  Pi  1 

Pl=-..=P„=- 

This  measure  does  not  take  into  account  the  individual  reliabilities  of  each  system 
component. 

Barlow-Proschan  (B-P)  Structural  Measure.  The  Barlow-Proschan  (B-P) 
stmctural  measure  assumes  that  component  reliabilities  are  not  known,  but  can  be 
assumed  to  be  the  same  for  each  component  and  assigned  the  value  p.  It  is  defined  by  the 
equation 

l'i;l,=jl[h(li,p)-h(Oi,p)]dp  (2) 

where  h(li,  p)  represents  the  system  reliability  function  when  component  i  is  functioning 
and  h(0i,  p)  represents  the  system  reliability  function  when  component  i  is  not  functioning 
[5:457]. 

Time-Dependent  Component  Importance  Measures.  While  stmctural  importance 
measures  are  only  dependent  upon  the  underlying  system  stmcture,  time-dependent 
measures  take  into  consideration  the  component  reliabilities  at  some  chosen  time  t.  They 
are  typically  utilized  when  both  the  system  stmcture  and  the  component  reliability 
functions  are  known.  Two  frequently  used  time-dependent  measures  include  one 
developed  by  Birnbaum  and  another  introduced  by  Veseley  and  Fussell. 

Birnbaum  Reliability  Importance  Measure.  Bimbaum’s  reliability  importance 
measure  assesses  a  component’s  importance  at  time  t.  If  a  system  is  comprised  of  n 
components  whose  reliabilities  at  time  t  are pi,  p2, ...,  and  h(pi,  p2,  ps, Pn)  represents 
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the  system  reliability  at  time  t,  then  the  Bimbaum  reliability  importance  measure  for 


component  i  is  given  by 


^  Mp) 
3  A 


(3) 


It  represents  the  decrease  in  system  reliability  when  component  i  fails  [6:266].  The 
Bimbaum  reliability  importance  measure  is  the  most  frequently  used  time-dependent 
measure  because  of  relative  ease  in  calculation  and  because  it  provides  the  ‘fairest’  basis  of 
comparison  between  components  [5:458]. 

Veseley-Fussell  (V-F)  Importance  Measure.  Another  popular  time-dependent 
component  importance  measure,  introduced  by  Veseley  and  Fussell  in  1972,  utilizes  cut¬ 
set  theory  to  define  component  importance.  The  V-F  importance  measure,  lyp  (t) , 
represents  the  conditional  probability  that  a  cut  set  containing  component  i  has  failed  at 
time  t,  given  that  the  system  has  failed  at  time  t. 

Many  other  time-dependent  measures,  most  of  which  are  variations  of  those  discussed 
previously,  also  exist.  For  the  sake  of  brevity,  these  additional  measures,  including  those 
developed  by  Butler  and  Aven  arising  from  network  theory,  will  not  be  discussed  in  this 
paper. 

Time-Independent  Component  Importance  Measures.  Both  structural  measures 
and  time-dependent  measures  have  inherent  characteristics  which  make  them  inappropriate 
for  certain  analyses.  Structural  measures  do  not  consider  component  reliabilities,  and 
time- dependent  measures  are  only  valid  for  one  specific  instance  in  time.  As  a  result. 
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time-independent  measures  have  been  developed  in  an  attempt  to  address  these  issues. 
Time-independent  measures  allow  component  importance  rankings  for  a  desired  time 
interval.  Several  time-independent  measures  have  been  developed,  most  of  which  are 
some  form  of  weighted  average  of  the  Bimbaum  reliability  measure  [7:160].  Two  of  the 
most  prominent  time-independent  measures  are  those  developed  by  Barlow  and  Proschan 
and  B.  Natvig. 

Barlow-Proschan  Time-Independent  Measure.  The  first  time-independent 
component  importance  measure  was  introduced  by  Barlow  and  Proschan  in  1975.  The  B- 
P  measure  represents  the  probability  that  component  i  causes  system  failure  in  the  time 
period  (0,  t).  It  is  represented  by 

(4) 

where  / (t)  represents  the  Birnbaum  reliability  measure  at  time  t  and  / (t)  is  the  failure 
probability  density  function  for  component  i.  Igp  can  also  be  interpreted  as  the 
probability  that  the  system  life  equals  the  life  of  component  i  [8:158]. 

Natvig  Importance  Measure.  In  1979,  Natvig  introduced  another  time- 
independent  component  importance  measure.  The  Natvig  measure  is  defined  by 

/n  =£  /f(0-P(o(0-(-lnp,)(rM  (5) 

where  /?(,)  (0  represents  the  reliability  function  for  component  i.  The  Natvig  measure 

represents  the  reduction  in  expected  remaining  system  lifetime  (up  to  time  t)  due  to  the 
failure  of  the  component  [9:280]. 
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Other  time-independent  measures  have  been  developed  by  Aven,  Bergman,  Narros, 
Boland,  and  Xie,  most  of  which  are  extensions  or  advancements  of  the  above  listed 
measures.  Furthermore,  a  significant  amount  of  work  has  been  done  in  the  development  of 
importance  measures  for  multi-state  and  repairable  components.  Space  does  not  allow 
discussion  of  these  additional  measures,  but  Boland  and  El-Neweihi  [5]  is  an  excellent 
reference  providing  an  overview  of  each  method  and  a  list  of  applicable  references. 

Numerical  Example  of  Component  Importance  Measures.  To  further  demonstrate 
the  calculation  of  the  various  importance  measures,  a  numerical  example  is  offered.  For 
the  given  structure  shown  in  Figure  5,  the  Bimbaum  stmctural  measure,  Bimbaum 
reliability  time-dependent  measure,  and  the  Barlow-Proschan  and  Natvig  time-independent 
measures  will  be  calculated. 


Figure  5.  Example  System 


Table  1  defines  the  probability  distribution  and  reliability  functions  for  the  various  system 
components. 
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Table  1.  Component  Failure  Distributions  and  Reliability  Functions  for  Example  System 


Component 

(i) 

Failure  Distribution 

f’(i) 

■■ 

1 

Weibull:  Shape  =1.1  (hrs) 
Scale  =  3500 
Location  =  0 

1.1- 

3500'"  ^’  ^ 

L  3500  J 

2 

Weibull:  Shape  =  1.1 

Scale  =  3500 
Location  =  0 

1.1 

3500^^  '^  ^ 

L  3500 

3 

Weibull:  Shape  =  1.5 

Scale  =  2000 
Location  =  0 

1.5- 

2000'^^^  ^ 

L  2000 

4 

Weibull:  Shape  =1.5 

Scale  =  2000 
Location  =  0 

1.5- 

2000'*^^  ^ 

L  2000 

5 

Weibull:  Shape  =  2.0 

Scale  =  2000 
Location  =  0 

^2000^  1 

2000'" 

Based  on  the  stracture  function,  the  system  reliability  function  is 

/*(p)  =  [i-(i-a)-(i-F2)]-[i-(i-F3)(i-P4)]-F5  (6) 

Birnbaum  Structural  Measure  Example.  Since  both  components  1  and  2  as  well 
as  3  and  4  are  identical  and  in-parallel  (and  the  stmctural  importance  measure  does  not 
consider  component  reliability),  the  stmctural  importance  measure  values  for  components 
1  through  4  will  be  the  same. 

Recall  from  equation  (1)  that 


d  h(p) 


dp, 


pi=-=Pn= 


For  component  1, 
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=  (1- /’2)- ft)- (1-/^4)]- /^5 


(7) 


Therefore,  the  Birnbaum  structural  measure  component  ranking  (in  descending  order)  is 
5,  {1,2,  3,4). 

Birnbaum  Reliability  (Time-Dependent)  Measure  Example.  Recall  from 

equation  (3),  I g\t)  =  ^  .  Since  this  is  a  time-dependent  measure,  a  specified  time 

dp. 


value  (0  must  be  selected.  In  this  example,  t  =  1000  hours.  Therefore, 
for  component  1 , 
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I?  it)  =  =  (1  -  (0)  •  [1  -  (1  -  P3  (0)  •  (1  -  p,  (0)]  •  Ps  it) 


(9) 


=  .158135 

where  pi{t}  is  given  in  Table  1.  Since  component  1  and  2  are  identical  and  in-parallel, 
component  2’s  importance  measure  will  also  equal  .158135. 

Similarly,  for  components  3  and  4,  7®  (t)  =  ^  =  .220421  =  (t) . 

dPi 

For  component  5,  7 f  ^  (t)  =  ^  =  .866066 . 

dps 

Therefore,  the  Birnbaum  reliability  (time-dependent)  importance  measure  component 
ranking  (in  descending  order)  is  5,  {3, 4},  { 1, 2}. 

Barlow-Proschaii  Time-Independent  Measure  Example.  From  equation  (4), 

=  [  ^b'  (t)  •  (t)dt .  A  time  period  of  interest  (for  the  range  of  integration)  must 

be  specified  to  calculate  time-independent  measures.  In  this  example,  the  time  period  will 
be  0  to  50,000  hours  (i.e.  x  =  50,000).  For  components  1  and  2,  where  7^  ^  (t)  is  given  in 
equation  (9)  and  pi( t)  and/  (r)  are  provided  in  Table  1, 

=  J”  7«(O-/'‘'(Ofi??  =  .056671  =  7®  . 

Similarly,  for  components  3  and  4, 

I  BP  =  /f  (0  •  /  ®  it)dt  =  .145126  =  7  w  . 

For  component  5, 

...  (.50,000 

/BP=i  7f(0-/^'^0^^r  =  .596417  . 
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Therefore,  the  Bimbaum  time-independent  importance  measure  component  ranking  (in 
descending  order)  is  5,  {3, 4},  {1,2}. 

Natvig  Time-Independent  Importance  Measure  Example.  From  equation  (5), 

^  (0  •  P(i)  (t)  •(-  In  /?(,)  (ty)dt .  For  components  1  and  2,  where  I g  ^  (t)  is  given  in 
equation  (9)  and  pi(t)  is  provided  in  Table  1, 

...  fSO.OOO  . 

=1  /l^\?)-Pa)(0-(-lnP(,)(0¥^  =  66.7423  =  /®  . 

For  components  3  and  4, 

/•50,000  ..j,  ... 

In'  =  I  If  (0  ■  Po)  (0  •(-lnp(3)(0)dr  =  142.9822  =  /f 
and  for  component  5, 

rSO.OOO  ... 

If  =  I  lf(0-P(5fO<-lnp,,,(0)dt  =  4023612  . 

Therefore,  the  Natvig  importance  measure  component  ranking  (in  descending  order)  is 
5,  {3,4},  {1,2}. 

In  this  particular  example,  the  various  demonstrated  measures  resulted  in  equivalent 
importance  rankings  for  the  system  components  (the  Bimbaum  stmctural  method  did  not 
differentiate  between  components  {1,2}  and  {3, 4}  because  it  considered  only  system 
structure  and  not  component  reliability)  as  summarized  in  Table  2. 


Table  2.  Importance  Measure  Rankings  for  Example  System 


Importance  Measure 

Ranking  (highest  to  lowest) 

Bimbaum  Structural 

5,  {1,2,  3,4} 

Bimbaum  Reliability 

5,  {3,4},  {1,2} 

Barlow-Proschan 

5,  {3,4},  {1,2} 

Natvig 

5,  {3,4},  {1,2} 
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However,  due  to  the  different  methods  used  in  the  calculation  of  component  importance 
measures,  there  will  not  necessarily  be  agreement  in  component  rankings  between  the 
various  measures.  Several  instances  were  cited  in  the  literature  where  one  measure 
produced  completely  opposite  ranking  results  from  another  measure.  Therefore,  analyst 
judgment  is  required  for  the  selection  of  the  most  appropriate  importance  measure  for  any 
given  situation  [10:1431]. 

Simulation  Experimental  Design  and  Factor  Screening  Methods 

The  purpose  of  any  experiment  is  to  gain  insight  about  a  particular  system  [1 1:424]. 
Typically,  changes  are  made  to  particular  inputs  {csAltd  factors),  and  the  effects  of  these 
changes  on  some  output  parameter(s)  (called  responses)  are  analyzed  and  measured. 
Computer  simulation  models  allow  analysts  the  benefit  of  experimenting  with  a  system 
model  instead  of  the  actual  system.  This  usually  saves  time  and  money,  and  is  frequently 
the  only  practical  method  of  analyses. 

Rather  than  randomly  trying  different  combinations  of  input  factor  levels  to  ascertain 
their  affect  on  the  response,  designed  experiments  provide  an  efficient  and  systematic 
method  for  conducting  such  analysis.  Using  a  designed  approach,  the  analyst  can 
determine  in  advance  the  number  of  simulation  runs  and  input  configurations  for  each  run 
to  obtain  the  desired  information  about  the  system  [12:657].  When  more  than  just  a  few 
factors  are  under  study,  a  logical  first  step  is  to  determine  or  ‘isolate’  those  factors  which 
significantly  affect  the  response  measure.  The  literature  commonly  describes  this  as 
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factor  screening.  Several  methods  of  factor  screening  are  outlined  in  the  literature 
including  two-level  factorial  designed  experiments,  fractional  factorial  experiments,  and 
Plackett-Burman  (P-B)  designs.  Most  factor  screening  methods  consist  of  two-level 
designed  experiments  [13:50].  In  fact,  the  most  popular  two-level  experimental  designs 
are  fractional  factorials  and  P-B  designs  [14:94].  Not  until  recently  have  designed  factor 
screening  experiments  been  used  in  the  field  of  reliability  to  identify  important  factors 
which  affect  system  performance  [15:206]. 

A  P-B  designed  experiment  was  used  in  this  effort  to  identify  the  subset  of  active 
factors  which  affect  availability  estimation  accuracy.  This  section  provides  a  brief 
discussion  of  two-level  factorial  designed  experiments,  fractional  factorial  experiments,  as 
well  as  an  in-depth  discussion  of  P-B  designs  and  their  projection  properties. 

Two-Level  (2*)  Factorial  Designed  Experiments.  A  full  two-level  factorial 
experiment,  where  each  factor  is  assigned  a  high  and  low  level,  will  be  used  to  estimate  the 
effects  of  each  of  the  k  factors  under  study  as  well  as  their  interaction  effects.  It  requires 
simulation  runs  for  each  of  the  2*  possible  factor-level  combinations  (called  design  points) 
[12:660].  When  a  relatively  small  number  of  factors  are  under  consideration,  a  full  two- 
level  factorial  experiment  is  desirable  for  factor  screening  because  it  identifies  all  active 
effects  without  confounding.  However,  when  k  becomes  moderate  in  size,  which  is  most 
often  the  case,  the  amount  of  runs  required  can  become  unreasonably  large. 

Fractional  Factorial  Designs.  To  reduce  the  number  of  runs  required,  a  fractional 
factorial  experiment  can  be  run  using  a  subset  (2*"^)  of  the  2*  full-factorial  design  points. 
This  will  introduce  confounding,  thus  reducing  the  amount  of  conclusive  information 
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gained  from  the  experiment.  However,  since  we  commonly  assume  higher-order 
interactions  are  negligible  in  factor  screening  experiments  [16:17],  fractional  factorials  can 
serve  as  excellent  screening  designs  where  only  the  main  and  two-factor  interactions  are  of 
interest.  The  main  disadvantage  of  fractional  factorials  is,  like  full  factorials,  they 
frequently  require  an  impractical  amount  of  simulation  runs. 

Plackett-Burman  (P-B)  Experimental  Designs.  P-B  designs  have  traditionally  been 
used  in  factor  screening  experiments  to  identify  significant  main  effects  [17:137],  and  they 
require  significantly  fewer  runs  than  full  and  fractional  factorials.  P-B  designs  are 
designed  experiments  with  two  levels  for  estimating  the  effects  of  «  -  7  factors  at  two 
levels  in  n  runs.  The  number  of  runs  (n)  must  be  a  multiple  of  four  [18:423].  P-B  designs 
are  useful  for  screening  experiments  where  several  factors  are  of  interest,  but  only  a 
portion  of  these  factors  are  suspected  as  being  significant.  They  allow  analysis  of  the 
main  effects  with  a  minimal  number  of  experimental  runs.  The  aliasing  structure  of  P-B 
designs  is  complex,  with  the  main  effects  being  aliased  with  other  interaction  effects. 
Therefore,  P-B  designs  are  most  effective  when  the  experimenter  has  good  reason  to 
believe  that  the  interaction  effects  are  negligible.  However,  if  some  interaction  effects  are 
significant,  they  may  be  identified  when  using  the  P-B  projection  techniques  outlined  by 
Lin  and  Draper  in  [19]. 

Projection  Properties  of  P-B  Designs.  When  an  experimental  design  is  projected, 
analysis  is  conducted  in  a  smaller  dimension  factor  space  to  provide  more  detailed 
information  concerning  certain  retained  factors.  For  example,  let’s  say  an  initial  full 
factorial  experiment  was  conducted  assessing  four  factors  with  no  replicates  (i.e.  16  runs) 
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and  only  two  factors  proved  significant.  By  ignoring  the  two  insignificant  factors,  the 
design  could  be  projected  into  a full  factorial  experiment  with  four  replicates.  In  this 
example,  the  projection  produces  replicates  which  allow  for  the  calculation  of  pure  error 
and  the  assessment  of  the  appropriateness  of  the  model  fit. 

Because  of  the  saturated  nature  of  Plackett-Burman  designs,  their  projection  properties 
are  limited,  but  they  can  still  he  useful.  Myers  and  Montgomery  address  this  limitation  by 
describing  the  projection  properties  of  Plackett-Burman  (P-B)  experimental  designs  as 
unattractive  [20:170].  However,  with  augmentation  of  additional  runs  to  the  original  P-B 
design,  some  beneficial  projection  properties  can  be  obtained.  As  Lin  and  Draper  show, 
P-B  designs  can  be  quite  useful  in  conducting  screening  experiments  using  a  limited 
number  of  mns.  Additionally,  interaction  effects  can  be  analyzed  by  utilizing  Lin  and 
Draper’s  P-B  projection  techniques  to  obtain  a  higher  resolution  design  in  the  significant 
factor  space. 

Lin  and  Draper’s  P-B  Projection  Techniques.  An  overview  of  Lin  and  Draper’s 
P-B  projection  concepts  can  he  summarized  in  a  few  concise  steps: 

(1)  Conduct  a  P-B  designed  experiment  with  the  appropriate  number  of  runs  (n) 
for  the  factors  which  are  to  be  screened  and  analyzed. 

(2)  Using  Yates  algorithm  [21:323-324],  identify  the  k  factors  which  exhibit 
significant  main  effects. 

(3)  Use  the  associated  P-B  design  columns  for  the  k  significant  factors  as  the 
projected  design  in  the  k  factor  dimension. 

(4)  If  necessary,  conduct  supplemental  experimental  runs  using  specified  levels 
for  the  k  significant  factors  to  achieve  a  desired  resolution  for  the  projected 
design. 
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P-B  Projections.  Table  3  delineates  the  projections  identified  for  the  12-run 


Plackett-Burman  design. 

Table  3.  Projection  of  a  12-run  Plackett-Burman  Design  into  k  Dimensions  [19] 


k 

Description 

2 

2.1 

2^  design  with  3  replicates 

3 

3.1 

2  design  plus  2  '  design 

4 

4.1 

Add  one  more  run  to  obtain  a  2'*  ’  design 

Add  two  more  runs  to  obtain  3/4  replicate  design 

Add  five  more  runs  to  obtain  a  2^^  design 

5 

5.1 

Add  two  more  runs  to  obtain  a  design 

Add  six  more  runs  to  obtain  a  2y^  design 

5.2 

Add  two  more  runs  to  obtain  a  2/^“^  design 

Add  eight  more  runs  to  obtain  a  2^^  design 

Add  ten  more  runs  to  obtain  a  2^“^  design 

A  brief  theoretical  example  may  be  the  best  method  to  demonstrate  Lin  and  Draper’s  P-B 
projection  techniques.  The  following  is  an  example  where  n  =  12  and  k  =  3>.  After 
conducting  the  12  P-B  runs,  suppose  only  3  of  the  1 1  main  effects  prove  to  be  significant 
(i.e.  k  =  3).  By  focusing  only  on  the  3  columns  that  correspond  to  the  k  significant  factors 
(in  this  example  A,  B,  and  C),  the  smaller  design  can  be  decomposed  into  a  full  2^  design 
and  a  2^'^  design  (where  I  =  ±ABC).  Figure  6  shows  a  full  12-run  P-B  design.  If,  after 
conducting  the  12  runs  for  the  P-B  design,  only  factors  A,  B,  and  C  possess  significant 
main  effects,  the  design  can  be  projected  (with  rows  rearranged)  into  the  arrangement 
shown  in  Figure  7. 
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Figure  6.  Plackett-Burman  Design  («  =  12) 
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Figure  7.  P-B  Design  Projection  forn  =  12  and  ^  =  3  (A,  B,  C) 

As  can  clearly  be  seen,  runs  1  through  8  represent  a  full 't  design,  and  runs  9  through  12 
represent  a  fractional  design  (where  I  =  -ABC).  These  12  runs  will  estimate  all  main 
effects  of  the  3  selected  factors  without  aliasing  and  will  also  provide  information  to 
calculate  pure  error  needed  for  lack  of  fit  testing  [19]. 

When  k  =  A  and  /:  =  5  for  the  12-run  P-B  design,  no  complete  projection  exists  for  the 
factors  of  interest.  However,  viable  projections  can  be  achieved  by  conducting 
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supplemental  runs.  When  k  =  A,  one  run  can  be  added  to  obtain  a  2^^  design,  or  five  runs 
can  be  added  to  obtain  a  full  factorial  design.  An  additional  option  is  to  supplement  the 
runs  to  project  the  design  into  a  three-quarter  replicate.  The  three-quarter  replicate 
consists  of  fewer  runs  than  a  full  factorial  design  but  more  rans  than  a  half  fraction.  The 
three-quarter  replicate  allows  for  estimation  of  the  main  effects  and  2-factor  interactions 
without  aliasing  with  other  2-factor  interactions  [22].  For  k  =  A,  two  additional  runs  are 
needed  to  complete  a  three-quarter  fraction  design  for  the  4  factors  of  interest.  When 
k  =  5,  two  possible  projection  opportunities  occur  depending  on  the  structure  of  the  rows 
of  the  5  selected  columns  from  the  original  P-B  design.  If  a  repeat-run  pair  emerges,  Lin 
and  Draper  call  this  a  5. 1  design,  where  two  more  runs  can  be  added  to  obtain  a  2]'^^ 
design,  and  six  more  runs  can  be  added  to  obtain  a  2^"^  design.  If  a  mirror  image  pair 
emerges  from  the  selected  columns  of  the  P-B  design,  this  is  a  5.2  design,  where  two 
additional  runs  gives  a  2^^^  design,  eight  additional  runs  gives  a  2^'  design,  and  ten 
additional  runs  achieves  a  2^“^  design. 

Benefits  of  P-B  Designs.  Utilizing  Plackett-Burman  designs  and  Lin  and  Draper’s 
projection  techniques  offer  an  efficient  way  to  conduct  screening  experiments  when  many 
factors  are  being  considered,  only  a  few  are  suspected  of  being  significant,  and  higher 
order  effects  are  assumed  to  be  negligible.  The  projection  techniques  outlined  allow 
analysis  of  the  two-factor  interactions  in  the  A:-dimensions  of  the  projection  while  requiring 
less  additional  runs  than  a  standard  foldover. 
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Using  a  P-B  experimental  design  for  factor  screening  in  this  research  provided  the 
benefit  of  accomplishing  the  required  objectives  with  maximum  efficiency.  In  the  final 
experiment,  nine  input  data  characterization  factors  were  assessed  for  significance.  A 
substantial  amount  of  effort  was  required  to  set  up  each  experimental  run.  The 
completion  of  a  full  two-level  factorial  experiment  would  have  required  512  runs,  while 
any  viable  fractional  factorial  design  would  also  have  required  a  large  amount  of  runs. 

This  was  well  beyond  the  scope  of  this  research.  On  the  other  hand,  the  selected  P-B 
design  required  only  12  experimental  runs,  while  still  providing  analysis  of  the  main  effects 
and  some  two-factor  interactions. 

Past  Research 

The  literature  was  reviewed  for  research  in  the  areas  of  input  data  characterization  and 
factor  screening  for  system  availability  estimation.  Numerous  examples  of  factor 
screening  experiments  were  found  in  the  current  literature.  A  few  articles  reviewed  were 
closely  related  to  this  research  and  many  facets  of  the  final  experimental  design  were 
extracted  from  these  specific  efforts.  This  section  will  briefly  discuss  six  articles  which 
closely  paralleled  and/or  helped  formulate  the  methodology  for  this  thesis. 

Sensitivity  Analysis  of  Availability  Estimates.  Wolf  [23]  assessed  the  sensitivity  of 
space  system  availability  estimates  to  the  underlying  component  reliability  estimates.  He 
utilized  an  iterative  response  surface  methodology  (RSM)  to  identify  the  system 
components  whose  component  reliability  significantly  affected  average  system  availability 
estimates.  Individual  j^omponent  reliabilities  were  perturbed  to  high  and  low  levels,  and 
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fractional  factorial  experiments  were  used  for  factor  screening.  From  this  analysis,  Wolf 
formulated  a  regression  model  predicting  average  system  availability  regressed  against  the 
estimated  component  reliabilities.  Extensive  regression  analysis,  involving  several 
iterations,  was  necessary  to  identify  the  significant  or  ‘important’  components.  Four  of 
the  initial  one  hundred  components  were  retained  in  the  final  system  availability  regression 
model.  Wolf  found  very  little  sensitivity  of  predicted  system  availability  to  individual 
component  failure  rate  estimates.  He  surmised  that  this  insensitivity  may  be  due  in  part  to 
the  simplicity  of  the  model  [24:69]. 

Availability  Analysis  Using  Simulation.  Edgar  and  Bendell  [24]  tested  the 
robusmess  of  Markov  models  in  estimating  mean-time-to-failure  (MTTF),  mean-time-to- 
repair  (MTTR),  mean-time-to- first-failure  (MTTFF),  and  availability  for  coherent  systems 
of  identical  repairable  components  (up  to  10)  by  use  of  simulation.  Using  Weibull 
distributions  to  define  component  failure  and  repair  rates,  the  authors  analyzed  steady- 
state  simulation  versus  Markov  analytical  results  for  both  increasing  failure  rate  (IFR)  and 
decreasing  failure  rate  (DFR)  component  fadiure  and  repair  distributions.  In  general,  the 
simulation  steady-state  and  Markov  model  results  were  found  to  be  consistent.  The 
authors  concluded  that  failure  distributions  (as  opposed  to  repair  distributions)  were  more 
critical  in  defining  overall  system  behavior,  and  that  decreasing  failure  rates  were  more 
critical  than  increasing  failure  rates  [24:125]. 

System  Complexity  (or  Size).  Hwang,  Tillman,  and  Lee  [25]  performed  a  literature 
review  of  works  which  evaluate  reliability  calculation  methods  for  complex  systems.  Their 
definition  of  a  complex  system  was  one  that  could  not  be  categorized  as  a  series-parallel 
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structure.  They  categorized  these  complex  systems  as  either  small  (1-6  components), 
moderate  (7-9  components),  or  large  (10  or  more  components).  The  article  provided 
diagrams  of  the  chosen  example  complex  systems  for  the  study  with  some  small,  some 
moderate,  and  some  large.  They  applied  various  methods  defined  in  the  literature  to 
evaluate  the  reliability  of  each  example  complex  system.  Hwang,  Tillman,  and  Lee’s 
definitions  of  complexity/size  were  utilized  in  this  research  effort. 

Constant  Failure  Rate  Assumption.  A  common  practice  in  reliability  analysis  is  to 
assume  that  time  between  failure  follows  an  exponential  distribution  (i.e.  a  constant  failure 
rate).  Mortin,  Krolewski,  and  Cushing  [26]  provided  examples  where  this  assumption 
produced  erroneous  results.  They  concluded  that  indiscriminate  use  of  this  simplifying 
assumption  can  introduce  significant  error  in  the  analysis  [26:54]. 

Repair  Distributions.  Kline  [27],  through  in-depth  analysis  of  several  systems, 
verified  that  the  lognormal  is  a  good  distribution  for  describing  repair  rates.  He  also 
concluded  that  use  of  the  exponential  distribution  for  repair  rates  resulted  in  negligible 
error  when  the  true  underlying  repair  distribution  was  lognormal  [27:79]. 

Comparison  of  Screening  Designs  for  Simulation  Models.  Webb  and  Bauer  [28], 
using  a  large-scale  computer  simulation,  compared  three  methods  of  analysis  for  a 
Plackett-Burman  screening  design:  the  Box  and  Meyer  approach,  the  traditional  response 
surface  methodology  (RSM)  approach,  and  the  Hamanda  and  Wu  approach.  This  thesis 
employed  the  RSM  and  Box-Meyer  analysis  methods. 

Box-Meyer  Bayesian  Method.  The  Box-Meyer  method  entails  deriving  the 
marginal  posterior  probability  that  a  factor  is  active  (i.e.  statistically  significant)  using 
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Bayesian  techniques.  This  method  determines  which  model  best  fits  the  data  by  examining 
all  possible  hypotheses  and  is  analogous  to  all-subsets  regression  [28:307],  Box  and 
Meyer  explain  their  method  as  follows: 

“The  Bayesian  approach  to  model  identification  is  as  follows.  We  consider  the  set  of  all 
possible  models  labeled  Mo, ...,  M„.  Each  model  Mi  has  an  associated  vector  of 
parameters  0,,  so  that  the  sampling  distribution  of  datay,  given  the  model  M„  is  described 
by  the  probability  density  f(y\Mi,  0,).  The  prior  probability  of  the  model  Mi,  is  piMi),  and 
the  prior  probability  density  of  0,  w/(0,-  IM,).  The  predictive  density  of  y,  given  model  M,, 
is  written /(ylMj),  and  is  given  by  the  expression 

/()’IM,)  =  J/(ylM,,0,)d0, 

where  /?,•  is  the  set  of  possible  values  of  0;.  The  posterior  probability  of  the  model  M„ 
given  the  data  y,  is  then 

A=0 

The  posterior  probabilities  p(Af,l  y)  provide  a  basis  for  model  identification.  Tentatively 
plausible  models  are  identified  by  their  large  posterior  probability”  [14:95]. 

Since  it  considers  the  possibility  of  interactions,  the  Box-Meyer  method  increases  the 

likelihood  of  identifying  active  factors.  This  is  “particularly  true  of  Plackett-Burman 

designs  where  the  number  of  runs  is  not  a  power  of  two”  [14:94]. 

Response  Surface  Methodology  (RSM).  The  RSM  approach  consists  of 

examining  the  magnitude  of  the  main  effects,  using  analysis  of  variance  (ANOVA),  and 

examining  normal  probability  and/or  Pareto  plots.  A  Pareto  plot  is  a  bar  chart  where  the 

length  of  the  bars  is  proportional  to  the  absolute  value  of  the  estimated  effects  [28:309]. 


Summary 

A  key  objective  of  this  research  was  to  ascertain  whether  there  is  utility  in  focusing  on 
‘important’  components  when  characterizing  input  data  for  availability  models.  This 
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chapter  provided  a  detailed  review  of  current  methods  for  computing  component 
importance.  Additionally,  a  general  overview  of  two-level  screening  designs  as  well  as  a 
thorough  review  of  Plackett-Burman  (P-B)  designs  was  provided.  A  P-B  screening 
experimental  design  was  used  in  this  thesis  to  determine  which  selected  characterization 
factors  were  significant.  Finally,  pertinent  literature  which  shaped  the  methodology  for 
this  effort  was  discussed. 

Many  factors  contribute  to  the  accuracy  of  availability  models.  In  an  effort  to 
supplement  the  analyst  interviews,  the  literature  review  helped  identify  input  data 
characterization  factor  candidates  for  analysis:  component  importance,  underlying 
component  failure  and  repair  distribution  characteristics  (IFR  versus  DFR),  system 
stmcture  type,  and  system  complexity  level  (or  size).  The  literature  review  also  provided 
insight  into  appropriate  factor  levels  for  the  two-level  screening  experiments  and 
applicable  analysis  methods. 
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III.  METHODOLOGY:  PRELIMINARY  EXPERIMENT 


General  Methodology  Overview 

The  general  methodology  for  this  research  entailed  a  designed  screening  experiment  to 
identify  significant  input  data  characterization  factors  affecting  availability  estimate 
accuracy.  The  RSM  and  Box- Meyer  methods  discussed  previously  were  used  for  analysis 
of  the  experimental  output  data.  The  research  was  done  in  two  steps:  a  simplified 
preliminary  experiment  analyzing  five  factors  to  validate  and  refine  the  methodology,  and  a 
final  experiment  analyzing  nine  factors. 

Component  input  data  characterization  factors  of  interest  were  identified  using 
reliability  analyst  interviews,  ideas  derived  from  the  literature  review,  as  well  as  personal 
judgment.  The  nine  factors  identified  for  analysis  are  listed  in  Table  4. 

Table  4.  Selected  Experimental  Factors 


_ Input  Data  Characterization  Factors _ 

True  Failure  probability  density  function  (pdf)  of  important 

_ components _ 

True  Failure  probability  density  function  (pdf)  of  non-important 

_ components _ 

Number  of  data  points 

_ (assumed  to  be  same  for  all  components) _ 

Fitting  technique  for  Failure  pdf  of  important  components 
Fitting  technique  for  Repair  pdf  of  important  components 
Fitting  technique  for  Failure  pdf  of 

_ non-important  components _ 

Fitting  technique  for  Repair  pdf  of 

_ non-important  components _ 

_ System  Complexity  Level  (Size) _ 

_ System  Structure  Type _ 
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For  the  conduct  of  the  two-level  screening  experiments,  two  levels  for  each  factor  were 
selected,  labeled  high  and  low  for  simplicity.  Availability  models  for  various  generic 
systems  of  components  were  created  using  a  PC-based  RM&A  software  program 
developed  by  the  Headquarters  Air  Force  Operational  Test  and  Evaluation  Center 
(HQ  AFOTEC).  Each  system  of  components  was  designed  by  the  researcher  for  complete 
experimental  control  and  do  not  represent  actual  existing  systems.  In  accordance  with  the 
experimental  design,  factors  were  set  to  the  appropriate  levels  for  each  design  point.  The 
response  measure  for  each  simulation  run  was  system  availability  absolute  estimation 
error.  Following  the  simulation  runs,  the  responses  were  analyzed  to  screen  the  active 
factors  via  traditional  RSM  as  well  as  Box-Meyer  statistical  analysis  techniques. 

Preliminary  Experiment 

To  validate  the  general  methodology  and  to  expose  potential  problem  areas,  an  initial 
smaller  scale  screening  experiment  was  performed  on  a  subset  of  the  factors  listed  above. 
A  2^"^  factorial  designed  experiment  was  conducted  to  determine  which  of  5  input  data 
characterization  factors  (for  a  simple  series-parallel  structure)  might  significantly  affect 
availability  model  accuracy. 

Definitions.  The  system  considered  in  the  preliminary  experiment  was  a  coherent 
system  comprised  of  binary  state  components.  As  defined  previously,  a  coherent  system  is 
one  in  which  all  components  are  relevant  in  maintaining  a  functional  system.  Binary  state 
components  have  just  two  states:  functioning  or  failed.  The  states  are  represented  as 
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X(t)  =  1  if  the  component  functions  at  time  t  and 
X(f)  =  0  if  the  component  is  failed  at  time  t. 

A  system’s  (as  opposed  to  component)  reliability  function  is  depicted  as  h(p),  where  p 
represents  the  component  reliability  vector.  System  availability  (Ao)  is  defined  as  the 
percentage  of  time  the  system  will  perform  its  specified  function  (i.e.  in  operational 
condition)  in  a  given  period  of  time  [29:253]. 

Assumptions.  The  following  assumptions  were  applied  to  the  preliminary  experiment: 

(1)  The  structure  is  coherent  consisting  of  binary  state  components. 

(2)  All  component  failure  and  repair  distribution  means  are  bounded  by  the 
following  limits: 

(a)  Weibull  failure  distributions:  1000  <  mean  <  5000  (hours) 

(b)  Lognormal  repair  distributions:  10  <  mean  <  200  (hours). 

(3)  Only  these  specific  distributions  (Weibull  and  Lognormal)  are  used  to 
represent  the  tme  component  failure  and  repair  distributions. 

(4)  All  parallel  components  are  identical. 

(5)  No  negative  location  parameters  are  allowed  in  distribution  data  fitting. 

(6)  Distributional  fitting  results  obtained  for  identical  parallel  components  require 
only  one  set  of  input  data  sampled  from  one  component. 

(7)  Maximum  Likelihood  Estimation  (MLE)  methods  are  used  to  calculate  fitted 
distribution  parameters. 

(8)  The  response  function,  defined  as  the  absolute  error  of  the  system  availability 
measure  from  each  simulation  run,  is  approximately  linear  with  respect  to  the  input 
variables. 

(9)  Higher  order  interaction  effects  are  negligible. 

(10)  The  component  with  the  highest  ranking  Barlow-Proschan  time-independent 
importance  measure  represents  the  most  important  system  component. 
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Software.  The  software  used  to  create  the  availability  simulation  model  is  a  PC-based 
program  entitled  Rapid  Availability  Prototyping  Tool  for  Testing  Operational  Readiness 
(RAPTOR),  written  by  the  Headquarters  Air  Force  Operational  Test  and  Evaluation 
Center  (HQ  AFOTEC).  RAPTOR  can  be  used  to  create  availability,  reliability, 
maintainability,  and  sparing  models  for  various  systems  undergoing  operational  testing  and 
evaluation  (OT&E).  The  program  was  written  in  MODSIM II,  an  object-oriented 
simulation  language,  and  requires  the  user  to  graphically  define  the  system  Reliability 
Block  Diagram  (RED).  Component  failure  and  repair  rates  are  simulated  over  time  to 
determine  overall  system  R  &  M  characteristics  [30].  Weibull++  Version  4.0  was  the 
software  used  to  generate  and  fit  component  failure  and  repair  data  sets.  Weibull-f-t- 
Version  4.0  is  a  reliability  software  program  created  by  ReliaSoft,  Inc.  which  has  robust 
data  generation  and  fitting  routines  for  common  reliability  distributions  [31]. 

Design  of  Preliminary  Experiment.  The  structure  studied  was  a  simple  series- 
parallel  structure  consisting  of  five  components  depicted  in  Figure  8. 


Figure  8.  Experimental  Structure  for  Preliminary  Experiment 
The  experiment  consisted  of  a  2^“^  factorial  design  (with  three  replicates)  on  the  five  component 
series-parallel  system  shown  in  Figure  8.  Since  this  is  a  resolution  V  design,  the  main  effects  and 
two-factor  effects  can  be  estimated  without  aliasing  with  each  other.  However,  two-factor 
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interactions  are  confounded  with  three-factor  interactions  [32:163].  The  associated  experimental 
factors  and  levels  are  depicted  in  Table  5. 

Table  5.  Experimental  Factors  and  Levels  for  Preliminary  Experiment 


Factors 

Levels 

Number  of  data  points 

50 

+ 

A 

(assumed  to  be  same  for  all  components) 

10 

- 

Fitting  technique  for  Failure  pdf  of 

Weibull4-+  Top  MLE  Ranking 

-H 

B 

important  components 

Weibull++  MLE:  Exponential 

- 

Fitting  technique  for  Repair  pdf  of 

Weibull++  Top  MLE  Ranking 

+ 

C 

important  components 

Empirical 

- 

Fitting  technique  for  Failure  pdf  of  non- 

Weibull++  Top  MLE  Ranking 

-1- 

D 

important  components 

WeibullH-+  MLE:  Exponential 

- 

Fitting  technique  for  Repair  pdf  of  non- 

Weibull4-+  Top  MLE  Ranking 

-1- 

E 

important  components 

Empirical 

- 

The  Weibull++  Monte  Carlo  data  generation  module  was  used  to  generate  simulated 
failure  and  repair  times  from  the  defined  component  distributions.  The  Weibull++ 
distribution  wizard  was  used  to  fit  theoretical  distributions  to  the  generated  data  set  and  to 
calculate  distribution  parameters  using  the  maximum  likelihood  estimation  (MLE)  method. 
A  ‘forced-fit’  exponential  distribution  was  used  for  the  low  level  for  component  failure 
data  fitting  due  to  the  frequent  use  of  the  exponential  assumption  in  component  failure 
analysis.  Separate  data  sets  were  generated  and  fitted  for  each  of  the  three  replications. 

The  defined  system  failure  and  repair  distributions  as  well  as  the  (replication  1)  fitted 
distributions  for  each  component  are  listed  in  Tables  6  and  7. 
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Table  6.  System  Failure  True  and  Fitted  Distributions  (Replication  1) 


10  Data  Points 

50  Data  Points 

Component 

True  Failure 
Distribution 

Wizard  Fit 

Exponential 

Fit 

Wizard  Fit 

Exponential 

Fit 

1/2 

Weibull:  (hrs) 
Shape  =  1.1 
Scale  =  3500 
Location  =  0 

Weibull: 

Shape  =1.142 
Scale  =  3677 
Location  =  0 

Exponential: 

Mean  =  3333 
Location  =  0 

Weibull: 

Shape  =  1.304 
Scale  =  4018 
Location  =  0 

Exponential: 

Mean  =  3333 
Location  =  8.4 

3/4 

Weibull: 

Shape  =  1.5 
Scale  =  2000 
Location  =  0 

Normal: 

Mean  =  1284 

St  Dev  =  771 

Exponential: 

Mean  =  1250 
Location  =  14.2 

Weibull: 

Shape  =  1.212 
Scale  =  1663 
Location  =  99.7 

Exponential: 

Mean  =  1429 
Location  =  136.4 

5 

Weibull: 

Shape  =  2.0 
Scale  =  2000 
Location  =  0 

Weibull: 

Shape  =  1.872 
Scale  =  2014 
Location  =  0 

Exponential: 

Mean  = 1428 
Location  =  384.5 

Weibull: 

Shape  =  2.220 
Scale  =  2155 
Location  =  0 

Exponential: 

Mean  =  1429 
Location  =  478.9 

Table  7.  System  Repair  True  and  Fitted  Distributions  (Replication  1) 


10  Data  Points 

50  Data  Points 

Component 

True  Repair 
Distribution 

Wizard  Fit 

Low  Level 
Fit 

Wizard  Fit 

Low  Level 
Fit 

1/2 

Lognormal: 

Mean  =  40  (hrs) 

St  Dev  =  10 

Lognormal: 

Mean  =  43.4 

St  Dev  =  6.5 

Empirical 

Lognormal: 

Mean  =  39. 1 

St  Dev  =  8.8 

Empirical 

3/4 

Lognormal: 

Mean  =  70 

St  Dev  =  15 

Weibull: 

Shape  =  10.73 

Scale  =  65.2 
Location  =  0 

Empirical 

Lognormal: 

Mean  =  70.6 

St  Dev  =  16.5 

Empirical 

5 

Lognormal: 

Mean  =  60 

St  Dev  =  8 

Weibull: 

Shape  =  1.582 

Scale  =  20.0 
Location  =  38.9 

Empirical 

Weibull: 

Shape  =  2.744 

Scale  =  25.2 
Location  =  38.3 

Empirical 

Since  components  1  and  2  as  well  as  3  and  4  were  identical,  the  same  data  fit  was  used  for 
each  identical  pair.  Graphical  examples  of  the  results  for  failure  and  repair  pdf  data 
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fittings  for  component  5  are  shown  in  Figures  9  and  10.  The  generated  data  sets  for  the 
preliminary  experiment  data  fittings  are  available  in  Appendix  C. 


0  1000  2000  3000  4000  5000 


Figure  9.  Component  5  True  Failure  pdf  versus  Weibull++  wizard 
and  exponential  fits  (Replication  1  using  10  data  points) 


40  60  80  100 

t 


Figure  10.  Component  5  True  Repair  pdf  versus  Weibull++ 
wizard  fit  (Replication  1  using  10  data  points) 
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Simulation  Runs.  Run  duration  for  each  replication  was  50,000  hours  in  simulated 
time.  Three  replications  were  conducted  at  each  of  the 't  design  points,  resulting  in  48 
total  runs.  The  response  variable  was  defined  as  the  absolute  error  of  the  system 
availability  measure  from  each  simulation  run.  The  value  representing  true  availability 
(Ao  =  96.6355  %)  used  for  calculation  of  absolute  error  was  obtained  by  conducting  2000 
runs  using  the  defined  component  failure  and  repair  distributions.  Banks,  Carson,  and 
Nelson’s  [33:449]  formula  was  used  to  calculate  the  initial  estimate  of  the  number  of  runs 
needed  to  obtain  a  95%  confidence  limit  and  a  ±  .015%  tolerance  for  the  ‘true’  system 
availability  measure: 

(10) 

e 

where  R  is  the  estimated  number  of  runs  needed.  So  is  the  standard  deviation  of  the  initial 
sample,  and  £  is  the  desired  tolerance. 

Since  each  run  represents  independent  and  identically  distributed  random  variables, 
traditional  statistical  methods  apply.  One  hundred  initial  runs  of  50,000  hours  duration 
were  completed  resulting  in  an  So  for  Ao  of  .3168%.  From  equation  (10),  R  >  17 13.56 . 
Therefore,  1714  or  more  runs  were  necessary  to  obtain  a  baseline  availability  measure 
which  would  meet  the  specified  tolerance  of  ±  .015%  at  a  95%  confidence  level.  A  total 
of  2000  runs  were  completed  which  resulted  in  an  average  availability  value  (Ao)  of 
96.6355%.  This  point  estimate  of  system  availability  for  time  0  to  50,000  hours  was  the 
benchmark  of  comparison  to  calculate  the  absolute  error  of  the  system  availability  measure 
for  each  design  point  in  the  experiment. 
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Components  were  rank-ordered  by  their  Barlow-Proschan  time-independent 
importance  measure  for  0  to  50,000  hours,  where  component  5  was  deemed  the  most 
important  component.  Table  8  shows  the  calculated  B-P  importance  measure  values. 
Table  8.  Barlow-Proschan  Time-Independent  Importance  Measure  Values 


Component  (s) 

Calculated  B-P 
Importance  Measure 

1,2 

.056671 

3,4 

.145126 

5 

.596417 

Analysis  Methods  and  Software.  The  analyzed  multiple  regression  main-effects 
model  can  be  described  in  the  following  format: 

Yij=^o+  ^jXj  +  ^2X2  +  ^3X3+  PA+  P5X5+  e,y  (1 1) 

where  Fy  is  the  response  value  for  run  number  i  and  replication  j; 

po  represents  the  intercept  (or  response  mean); 

P*  represents  the  regression  coefficients  for  factors  k  =  1,...,5; 

Xk  represents  the  factor  level  (either  +1  or  -1)  for  factor  k;  and 
Eij  represents  the  residual  error  for  run  number  i  and  replication  j. 

Yates  algorithm  [21:323-324]  and  least  squares  methods  were  used  to  calculate  the  main 
and  interaction  effects.  The  correlation  coefficient  (R^),  ANOVA,  and  lack  of  fit  statistics 
were  calculated  to  assess  model  adequacy.  To  identify  significant  factors,  normal 
probability  plots,  Pareto  plots,  Box-Meyer  Bayes  plots,  and  linear  regression  coefficient 
t-test  statistics  were  used.  The  primary  analysis  software  was  IMP  version  3.1, 
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a  PC-based  statistical  analysis  program  developed  by  the  SAS  Institute.  IMP  possesses 
data  graphing,  experimental  design,  and  statistical  analysis  routines  [34:319-341]  which 
proved  very  useful  in  this  research. 
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IV.  RESULTS:  PRELIMINARY  EXPERIMENT 


Simulation  Results 

The  2^“'  experimental  design  and  resulting  responses  for  the  preliminary  experiment 
are  shown  in  Table  9. 


Table  9.  Experimental  Design  and  Responses 


Factors 

Observed  Availability* 

Absolute  Error  (F) 

Design 

B 

B 

c 

D 

E 

Replication 

Replication 

Replication 

Replication 

Replication 

Replication 

Point 

■ 

1 

2 

3 

1 

2 

3 

1 

n 

D 

D 

D 

n 

96.8046% 

96.9903% 

95.8230% 

0.1691% 

0.3548% 

0.8125% 

2 

n 

D 

D 

n 

D 

96.7202% 

97.0266% 

95.8372% 

0.0847% 

0.3911% 

0.7983% 

3 

D 

D 

D 

D 

D 

96.6918% 

96.9858% 

95.7640% 

0.0563% 

0.3503% 

0.8715% 

4 

D 

D 

D 

D 

D 

96.6324% 

96.8985% 

95.7639% 

0.0031% 

0.2630% 

0.8716% 

5 

D 

D 

D 

D 

O 

96.7904% 

96.8941% 

95.9042% 

0.1549% 

0.2586% 

0.7313% 

6 

D 

D 

D 

a 

D 

96.7261% 

96.8518% 

95.9385% 

0.0906% 

0.2163% 

0.6970% 

7 

n 

n 

D 

D 

n 

96.6137% 

96.8172% 

95.8354% 

0.0218% 

0.1817% 

0.8001% 

8 

n 

D 

Q 

n 

D 

96.5398% 

96.7937% 

95.9377% 

0.0957% 

0.1582% 

0.6978% 

9 

D 

D 

D 

D 

D 

miTinna^m 

96.0124% 

96.3905% 

0.0919% 

0.6231% 

0.2450% 

10 

D 

D 

D 

n 

D 

96.0496% 

96.3695% 

0.0921% 

0.2660% 

11 

D 

D 

D 

D 

D 

96.6290% 

95.7957% 

96.2251% 

0.0065% 

0.4104% 

12 

n 

D 

D 

D 

D 

96.5982% 

95.8427% 

96.2454% 

0.0373% 

0.7928% 

0.3901% 

13 

D 

D 

D 

D 

n 

96.7642% 

95.9374% 

96.2951% 

0.1287% 

0.6981% 

0.3404% 

14 

D 

Q 

D 

D 

D 

96.7929% 

95.9976% 

96.4092% 

0.1574% 

0.6379% 

0.2263% 

15 

D 

D 

D 

D 

D 

96.6571% 

95.8386% 

96.2923% 

0.0216% 

0.7969% 

0.3432% 

16 

D 

D 

D 

D 

n 

96.8079% 

95.8528% 

96.2844% 

0.1724% 

0.7827% 

0.3511% 

*  2000  Run  ‘Truth’  Availability  =  96.6355% 


Note  that  all  system  availability  estimates  from  each  run  were  within  ±  .88%  of  the  defined 


trae  system  availability. 
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Statistical  Analysis 


A  summary  of  the  key  model  statistics  is  provided  in  Table  10. 

Table  10.  Preliminary  Experiment  Model  Statistical  Results 


Statistic 

Value 

Interpretation 

.004537 

Model  explains  virtually 
none  of  output  variability 

Whole  Model  F-test 
p-value 

.9991 

Model  as  a  whole 
is  not  significant 

Lack  of  Fit  F-test 
p-value 

1.0 

Linear  model  is  appropriate 
(no  curvature) 

The  model  statistics  show  that  the  defined  main-effects  model  explains  very  little  of  the 
response  variation  and  that  a  linear  model  is  appropriate  for  the  experimental  region.  A 
summary  of  the  calculated  factor  effects  and  statistics  is  shown  in  Table  1 1. 

Table  11.  Estimated  Effects  and  Statistical  Analysis 


Factor 

Effect 

Estimate 

Interpretation 

.37850% 

<.0001 

Significant  (mean  response) 

A 

-.00386% 

.9654 

Not  significant 

B 

-.02694% 

.7624 

Not  significant 

C 

.01933% 

.8282 

Not  significant 

D 

-.01871% 

.8336 

Not  significant 

E 

.00598% 

.9465 

Not  significant 

The  t-test  for  each  effect  estimate  indicates  that  only  the  mean  response  (regression 
model  intercept  term)  is  significant.  A  supplemental  listing  of  statistical  analysis 
outputs  for  the  preliminary  experiment  is  provided  in  Appendix  A. 
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Graphical  Analysis 

Figures  11, 12,  and  13  show  the  normal  probability,  the  Pareto,  and  the  Box-Meyer 
Bayes  plots. 


Normal  Quantile 


Figure  11.  Normal  Probability  Plot 


Term  Scaled  Estimate 
B  -0.0136113 

C  0.00976686 

D  -0.0094553 

E  0.00302122 

A  -0.0019517 


Figure  12.  Pareto  Plot  of  Scaled  Estimates 
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Term 

Estimate 

Prior 

Posterior 

A 

-0.0436317 

0.2000 

0.0244 

B 

-0.304292 

0.2000 

0.0256 

C 

0.21834656 

0.2000 

0.0250 

D 

-0.2113806 

0.2000 

0.0250 

E 

0.06754200 

0.2000 

0.0245 

Figure  13.  Box-Meyer  Bayes  Plot 

The  normal  probability  and  Bayes  plot  results  are  consistent  and  indicate  that  no  effects 
are  significant.  The  Pareto  plot  indicates  that  factors  B,  C,  and  D  explain  the  most 
variation,  but  since  the  amount  of  explained  variation  by  the  model  is  negligible  this  result 
has  little  significance. 


Additional  Analysis 

Upon  closer  inspection  of  the  absolute  error  responses  shown  in  Table  9,  it  was 
discovered  that  a  possible  blocking  effect  may  be  present  between  replications.  For 
example,  notice  (in  Table  9)  that  the  absolute  errors  in  replication  1  are  the  smallest  values 
in  all  cases.  To  address  this,  additional  data  analysis  was  conducted  on  models  which 
included  a  blocking  variable  as  well  as  other  response  measures:  error  and  squared  error. 
Table  12  contains  a  summary  of  the  possible  significant  factors  resulting  from  all  analyses 
on  the  preliminary  experimental  data. 

Table  12.  Significant  Factors  Assessing  Alternative 
Responses  and  a  Blocking  Variable 


Blocking 

Variable 

Response 

Absolute 

Error 

Error 

Squared  Error 

No 

None 

Possibly  C 

Possibly  A  &  C 

Yes 

None 

None 

A  and  possibly  C 
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Statistical  analysis  showed  that  the  blocking  variable  was  strongly  significant  with  all  three 


response  measures. 

With  the  additional  responses  (error  and  squared  error),  factors  A  (number  of  data 
points)  and  C  (fitting  technique  for  repair  pdf  of  important  component:  component  5), 
presented  themselves  as  possible  significant  factors.  However,  these  conclusions  are  not 
definitive  and  thus  were  addressed  again  in  the  final  experiment. 

Summary 

The  statistical  analysis,  using  absolute  error  as  a  response  measure,  supports  the 
hypothesis  that  there  are  no  significant  effects.  With  the  absolute  error  response,  no 
effects  were  shown  to  be  significant  in  the  t-tests,  and  the  normal  probability,  Pareto,  and 
Bayes  plots  revealed  no  clear  significant  factor  effects.  This  means  that  using  fewer  data 
points  (i.e.  10  versus  50)  and  less  aggressive  fitting  techniques  (i.e.  exponential 
assumption  for  failure  rates  and  use  of  empirical  repair  distributions)  on  important  as  well 
as  non-important  components  did  not  significantly  degrade  model  accuracy  for  this 
particular  stmcture. 

However,  introducing  a  blocking  variable  in  conjunction  with  two  alternative 
responses,  error  and  squared  error,  revealed  that  factors  A  and  C  may  be  significant. 
Therefore,  the  results  fi-om  this  experiment  are  inconclusive.  Further  analysis  is  required 
to  determine  conclusively  if  the  number  of  data  points  (factor  A)  and  the  fitting  technique 
for  repair  pdf  of  important  component  (factor  C)  are  significant. 
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V.  METHODOLOGY;  FINAL  EXPERIMENT 


Insights  Gained  from  Preliminary  Experiment 

While  the  preliminary  experiment  assessed  five  input  data  characterization  factors,  the 
final  experiment  assessed  nine  factors  Hsted  in  Table  4.  Several  insights  were  gained  from 
the  preliminary  experiment  which  helped  refine  the  methodology  for  the  final  experiment. 
After  reviewing  the  methodology  and  results  of  the  preliminary  experiment,  AFOTEC 
analysts  recommended  low  and  high  levels  of  5  and  25  for  the  ‘number  of  data  points’ 
factor  levels.  They  felt  that  levels  of  10  and  50  data  points  were  too  generous  based  upon 
their  experience  in  past  operational  availabiUty  analyses.  They  also  pointed  out  that  the 
mean-time-to-failure  (MTTF)  /  mean-time-to-repair  (MRT)  ratios  were  relatively  large  for 
all  five  components  of  the  experimental  strueture,  and  that  a  wider  range  of  ratios  may  be 
more  appropriate  for  future  experimental  designs.  It  was  also  pointed  out  that  frequently 
the  analyst  will  not  have  a  priori  knowledge  of  component  failure  behavior.  This 
information  is  normally  required  for  the  calculation  of  component  importance  measures, 
with  the  exception  of  stmctural  importance  measures.  An  additional  suggestion  was  to 
analyze  the  variability  of  several  availability  model  outputs  for  individual  runs.  This  was 
addressed  in  a  separate  study  conducted  using  multivariate  techniques  on  several 
RAPTOR  model  output  measures.  A  summary  of  the  study  is  provided  in  Appendix  G. 
Finally,  it  was  discovered  that  a  significant  amount  of  time  and  effort  was  required  to  set¬ 
up  the  experimental  rans,  which  included  component  failure  and  repair  data  point 
generation  and  fitting,  constmction  of  RAPTOR  models,  and  completion  of  ‘trath’  data 
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runs.  Since  the  required  effort  would  increase  dramatically  with  the  addition  of  4  more 
factors,  any  subsequent  experimental  screening  design  would  need  to  economize  on  the 
number  of  simulation  runs. 

Final  Experiment 

Assumptions.  To  produce  diversity  in  the  MTTF/MRT  ratios  for  the  system 
components,  wider  bounds  were  allowed  for  the  means  of  the  component  failure  and 
repair  distributions.  They  were  bounded  by  the  following  limits: 

(1)  Weibull  failure  distributions:  1000  <  mean  <  6500  (hours) 

(2)  Lognormal  repair  distributions:  50  <  mean  <  3000  (hours). 

The  most  important  components  in  a  stmcture  were  deemed  as  the  ones  which  fell  in  the 
top  20%  of  component  importance  measure  rankings  based  upon  component  failure 
distributions.  To  allow  for  the  calculation  of  the  importance  measures  without  knowledge 
of  the  underlying  failure  distributions,  the  Bimbaum  structural  importance  measure  was 
used.  This  measure  is  based  solely  upon  system  stmcture.  All  other  assumptions  outlined 
in  the  preliminary  experiment  also  applied  to  the  final  experiment. 

Structures.  20  components  were  designed  which  were  used  for  the  building  of  system 
stmctures  for  the  RAPTOR  models.  Each  component  was  designed  to  have  tme  Weibull 
failure  and  lognormal  repair  distributions  randomly  set  within  the  established  bounds  for 
the  distribution  means.  Increasing  failure  rate  (IFR)  and  decreasing  failure  rate  (DFR) 
configurations  were  created  for  each  component  while  maintaining  the  same  distribution 
mean.  To  accomplish  this,  randomly  selected  Weibull  shape  and  scale  parameters  were 
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utilized  to  create  the  IFR  failure  distributions.  Using  a  randomly  generated  DFR  shape 
parameter  for  each  component,  the  same  average  failure  rate  was  maintained  by  adjusting 
the  Weibull  scale  parameter  to  achieve  an  identical  mean  failure  rate  as  in  the  IFR 
configuration.  This  procedure  was  used  to  ensure  that  the  results  were  not  biased  by 
producing  a  different  average  failure  rate  when  reconfiguring  a  component  from  IFR  to 
DFR.  The  shape  parameters  ranged  from  1.1  to  4.0  for  IFR  configurations  and  from 
.4  to  .95  for  DFR  configurations.  A  complete  listing  of  component  failure  and  repair 
distribution  parameters  (for  both  configurations)  is  shown  in  Appendix  B. 

Four  basic  structures  were  created  from  the  set  of  20  components  described  above:  a 
small/series-parallel  structure,  a  small/complex  stmcture,  a  large/series-parallel  structure, 
and  a  large/complex  structure.  The  small  structures  used  components  1  through  5,  while 
the  large  structures  were  comprised  of  all  20  components.  Appendix  B  provides  reliability 
block  diagrams  for  each  structure. 

Design  of  Final  Experiment.  The  factors  and  levels  for  the  final  experiment  are 
shown  in  Table  13.  Since  each  run  demanded  a  large  set-up  effort,  a  design  which 
minimized  the  number  of  runs  was  preferable.  Replications  were  stiU  desired  to  increase 
the  confidence  in  the  results  and  to  estimate  pure  error  for  lack  of  fit  testing.  A  full 
factorial  experiment  would  require  1536  runs  (i.e.  512  *  3  replications),  and  a 
fractional  factorial  design  would  require  48  runs  (i.e.  16*3  replications).  A  Plackett- 
Burman  (P-B)  design  was  chosen  because  it  required  only  36  (i.e.  12*3  replications) 
total  simulation  runs  to  assess  the  nine  factors. 
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Table  13.  Factors  and  Levels  for  Final  Experiment 


Factors 

Levels 

A 

True  Failure  probability  density  function  (pdf)  of 

Weibull  IFR 

+ 

important  components 

Weibull  DFR 

- 

B 

True  Failure  probability  density  function  (pdf)  of 

Weibull  IFR 

+ 

non-important  components 

Weibull  DFR 

- 

C 

Number  of  data  points 

25 

+ 

(assumed  to  be  same  for  all  components) 

5 

- 

D 

Fitting  technique  for  Failure  pdf  of  important 

Weibull++  Top  MLE  Ranking 

+ 

components 

Weibull++  MLE:  Exponential 

- 

E 

Fitting  technique  for  Repair  pdf  of  important 

Weibull++  Top  MLE  Ranking 

+ 

components 

Empirical 

- 

F 

Fitting  technique  for  Failure  pdf  of 

Weibull++  Top  MLE  Ranking 

B 

non-important  components 

Weibull++  MLE:  Exponential 

B 

G 

Fitting  technique  for  Repair  pdf  of 

Weibull++  Top  MLE  Ranking 

B 

non-important  components 

Empirical 

B 

H 

System  Complexity  Level  (Size) 

Large  (20  components) 

Small  (5  components) 

H 

I 

System  Stracture  Type 

Series-Parallel 

Complex 

1 

The  12-run  9-factor  P-B  design  used  for  the  final  experiment  is  shown  in  Table  14. 


Table  14.  12-run  Plackett-Burman  Design  for  Final  Experiment 


Design 

Point 

A 

B 

C 

D 

E 

F 

G 

H 

I 

1 

+ 

+ 

+ 

+ 

-H 

+ 

+ 

+ 

+ 

2 

+ 

+ 

-1- 

+ 

- 

3 

- 

- 

+ 

- 

+ 

+ 

+ 

- 

- 

4 

+ 

- 

- 

+ 

- 

+ 

+ 

+ 

- 

5 

- 

+ 

- 

- 

+ 

- 

+ 

+ 

+ 

6 

- 

- 

+ 

- 

- 

+ 

- 

+ 

+ 

7 

- 

- 

- 

+ 

- 

- 

+ 

- 

+ 

8 

+ 

- 

- 

- 

+ 

- 

- 

-1- 

- 

9 

+ 

+ 

- 

- 

+ 

- 

- 

+ 

10 

+ 

+ 

+ 

- 

- 

- 

+ 

- 

- 

11 

- 

+ 

+ 

+ 

- 

- 

- 

+ 

- 

12 

+ 

- 

+ 

+ 

+ 

- 

- 

- 

+ 
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Distributional  Fittings.  As  in  the  preliminary  experiment,  Weibull++  was  used  to 
generate  and  fit  the  component  failure  and  repair  data  sets  for  each  configuration. 

Separate  generations  and  fits  were  conducted  for  each  replication.  Components  14, 15, 
and  16  as  well  as  18,  19,  and  20  were  identical  components,  therefore  only  one  generation 
and  fitting  was  conducted  for  each  triplicate  set  per  replication.  Final  experiment  fitting 
data  is  contained  in  Appendix  D  and  graphical  examples  for  the  fitted  distributions  for 
some  of  the  components  are  provided  in  Appendix  E. 

Important  Components.  A  complete  listing  of  the  Bimbaum  structural  component 
importance  measures  calculated  for  each  component  in  each  of  the  four  experimental 
structures  is  provided  in  Appendix  F,  with  a  summary  provided  in  Table  15. 

Table  15.  Top  20%  Important  Components 


Structure 

Top  20%  Important 
Components 

Small  /  Series-Parallel 

Component  3 

Small  /  Complex 

Component  1 

Large  /  Series-Parallel 

Components  4, 5,  13, 17 

Large  /  Complex 

Components  1,  4,  7,  8 

Simulation  Runs.  16  truth  runs  were  required  due  to  the  four  additional  factors.  For 
each  of  the  four  structures,  ‘truth’  runs  were  done  with  the  following  configurations: 

(1)  All  components  with  IFR  failure  distributions 

(2)  All  components  with  DFR  failure  distributions 

(3)  Important  components  with  IFR  failure  distributions  and  non-important 
components  with  DFR  failure  distributions 

(4)  Important  components  with  DFR  failure  distributions  and  non-important 
components  with  IFR  failure  distributions. 


50 


As  before,  each  simulation  run  duration  was  for  50,000  hours  simulation  time. 

Two  thousand  replications  were  run  to  establish  ‘truth’  availability  values  for  each 
configuration.  For  the  P-B  experimental  rans,  the  response  measure  was  again  the 
absolute  error  of  the  system  availability  measure  from  each  simulation  mn  as  compared  to 
the  ‘tmth’  measure. 

Analysis  Methods.  The  analysis  methods  were  identical  to  those  used  for  the 
preliminary  experiment.  Traditional  statistical  measures  were  used  to  assess  model 
adequacy,  and  normal  probability  plots,  Pareto  plots,  Bayes  plots,  and  linear  regression 
coefficient  t-test  statistics  were  used  to  identify  the  significant  factor  effects.  A  response 
surface  was  formed  to  graphically  portray  the  combined  affect  of  the  active  factors  on 
model  availability  estimation  error. 
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VI.  RESULTS:  FINAL  EXPERIMENT 


Simulation  Results 

The  results  from  the  truth  and  Plackett-Burman  experimental  RAPTOR  runs  for  the 
final  experiment  are  shown  in  Table  16. 


Table  16.  Numerical  Results  for  Final  Experimental  Runs 


Structure 

Component 
Failure  PDF 

Observed  Availability 

Absolute  Error  (F) 

Important  / 
Non-important 

Replication 

1 

Replication 

2 

Replication 

3 

Replication 

1 

Replication 

2 

Replica  tiot 
3 

1 

Large/ 

S-P 

IFR/IFR 

83.1373% 

81.0810% 

78.166% 

82.5297% 

2.0563% 

4.9713% 

0.6076% 

2 

Small  / 
Complex 

IFR/DFR 

77.3638% 

81.2591% 

78.2235% 

80.9242% 

3.8953% 

0.8597% 

3.5604% 

3 

DFR/DFR 

76.4428% 

77.3795% 

79.5758% 

74.8265% 

0.9367% 

3.1330% 

1.6163% 

■1 

Large/ 

Complex 

DFR/IFR 

60.4257% 

38.6057% 

61.4589% 

55.8074% 

21.820% 

1.0332% 

4.6183% 

m 

Large/ 

S-P 

IFR/DFR 

82.7799% 

80.6977% 

77.8648% 

71.5604% 

2.0822% 

4.9151% 

11.219% 

m 

Large/ 

S-P 

DFR/DFR 

81.6366% 

82.0842% 

76.8906% 

82.4661% 

0.4476% 

4.7460% 

0.8295% 

m 

Small/ 

S-P 

DFR/DFR 

64.6009% 

63.2109% 

65.2901% 

55.2021% 

1.3900% 

0.6892% 

9.3988% 

8 

Large/ 

Complex 

DFR/IFR 

60.4257% 

41.0340% 

61.8580% 

54.8932% 

19.391% 

1.4323% 

5.5325% 

9 

Small  / 
S-P 

IFR/IFR 

65.9448% 

64.6130% 

65.4925% 

64.7097% 

1.3318% 

0.4523% 

1.2351% 

10 

Small/ 

Complex 

IFR/IFR 

78.2001% 

76.4971% 

80.2750% 

78.4965% 

1.7030% 

2.0749% 

0.2964% 

11 

Large/ 

Complex 

IFR/DFR 

60.7345% 

60.7147% 

56.1168% 

60.1398% 

0.0198% 

4.6177% 

0.5947% 

12 

Small/ 

S-P 

DFR/IFR 

65.0705% 

63.5087% 

65.8172% 

65.2438% 

1.5618% 

0.7467% 

0.1733% 

A  much  larger  variability  in  the  response  was  observed  compared  to  the  preliminary 
experiment.  The  observed  absolute  errors  in  availability  estimates  ranged  from  .0198%  to 


21.82%. 
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Statistical  Analysis 


A  summary  of  the  key  model  statistics  is  provided  in  Table  17. 

Table  17.  Final  Experiment  Model  Statistical  Results 


Statistic 

Value 

Interpretation  | 

.333092 

HEBSiB 

Whole  Model  F-test 
p-value 

.2241 

Model  as  a  whole 
is  not  significant 

Lack  of  Fit  F-test 
p-value 

.9680 

Linear  model  is  appropriate 
(no  curvature) 

The  model  statistics  show  that  the  defined  main-effects  model  explains  approximately  one- 
third  of  the  response  variation  and  that  a  linear  model  is  appropriate  for  the  experimental 
region.  A  summary  of  the  calculated  factor  effects  and  statistics  is  shown  in  Table  18. 
Table  18.  Estimated  Effects  and  Statistical  Analysis 


Factor 

Effect 

Estimate 

Interpretation 

3.4997% 

.0001 

Significant  (mean  response) 

.89372% 

.5688 

Not  significant 

B 

-1.8335% 

.2471 

Not  significant 

C 

-3.5403% 

.0306 

Significant 

D 

-.04232% 

.9784 

Not  significant 

E 

.63297% 

.6861 

Not  significant 

F 

-.53829% 

.7310 

Not  significant 

G 

1.2852% 

.4142 

Not  significant 

H 

3.1045% 

.0555 

Significant 

I 

-1.5712% 

.3197 

Not  significant 

The  mean  absolute  error  of  availability  estimates  for  all  the  P-B  simulation  mns  is 
3.4997%.  The  t-test  for  each  effect  estimate  indicates  that  the  mean  response 
(regression  model  intercept  term),  factor  C  (number  of  data  points)  effect,  and 
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factor  H  (system  complexity/size)  effect  are  significant.  A  supplemental  listing  of 
statistical  analysis  outputs  for  the  final  experiment  is  provided  in  Appendix  A. 


Graphical  Analysis 

Figures  14,  15,  and  16  show  the  normal  probability,  the  Pareto,  and  the  Box-Meyer 
Bayes  plots. 


Normal  Quantile 

Figure  14.  Normal  Probability  Plot 


Term 

Scaled  Estimate 

C 

-1.7952429 

H 

1.5742576 

B 

-0.9297767 

I 

-0.7967495 

G 

0.6517153 

A 

0.4531999 

E 

0.3209727 

F 

-0.2729623 

D 

-0.0214613 

Figure  15.  Pareto  Plot  of  Scaled  Estimates 
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Term 

Estimate 

Prior 

Posterior 

A 

0.5770951 

0.2000 

0.0284 

B 

-1.1839578 

0.2000 

0.0467 

C 

-2.2860239 

0.2000 

0.2430 

D 

-0.0273283 

0.2000 

0.0244 

E 

0.4087198 

0.2000 

0.0263 

F 

-0.3475843 

0.2000 

0.0258 

G 

0.8298804 

0.2000 

0.0335 

H 

2.0046260 

0.2000 

0.1524 

I 

-1.0145637 

0.2000 

0.0393 

Figure  16.  Box-Meyer  Bayes  Plot 

The  normal  probability,  Pareto,  and  Bayes  plot  results  are  consistent  and  suggest  that 
factor  C  (number  of  data  points)  and  factor  H  (system  complexity/size)  are  significant, 
while  all  other  factors  are  not  significant. 

Significant  Effect  Model 

A  subsequent  regression  model  containing  only  factors  C,  H,  and  their  interaction  term 
was  analyzed  to  determine  if  the  C*H  interaction  term  was  significant.  The  results  are 
shown  in  Table  19. 

Table  19.  Estimated  Effects  and  Statistical  Analysis  for  C,  H,  C*H  Model 


Factor 

Effect 

Estimate 

Interpretation 

3.4997% 

<0001 

Significant  (mean  response) 

c 

-3.5403% 

.019 

Significant 

H 

3.1045% 

.0379 

Significant 

C*H 

-2.365766 

.1086 

Not  significant 

Statistic 

Value 

Interpretation 

.296981 

Model  explains  approximately 
one-third  of  output  variability 

Whole  Model 
F-test  p-value 

.0095 

Model  as  a  whole 
is  significant 
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In  this  case,  the  model  explained  approximately  30%  of  the  response  variability,  and  the 
model  as  a  whole  was  significant.  As  before,  the  main  effects  for  factors  C  and  H  were 
significant.  The  C*H  interaction  effect  was  not  significant  at  a  10%  level  of  significance 

Response  Surface 

A  response  surface  was  developed  for  the  resulting  C  and  H  main-effects  model: 

Y=  3.4997-  1.770133C-\-  1.5522389H  (12) 

where  Y  is  the  estimated  absolute  error  in  the  availability  estimate;  and 

C  and  H  represent  the  factor  level  (either  -i-l  or  -1)  for  each  factor. 

The  resulting  response  surface  and  contour  plot  are  shown  in  Figure  17. 

Response  Surface  i 

Error 

H 

H 

-1 

C 

Figure  17.  Two-Factor  Model  Response  Surface  and  Contour  Plot 
As  the  plots  in  Figure  17  demonstrate,  a  high  level  for  factor  C  (number  of  data  points) 
and  a  low  level  for  factor  H  (system  size)  result  in  the  smallest  availability  estimation 
error. 


Contour  Plot 


-1  1 
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Additional  Analysis 

As  with  the  preliminary  experiment,  subsequent  analysis  was  performed  using  error  and 
squared  error  as  response  measures  as  well  as  introducing  a  blocking  variable  for  the 
replications.  In  all  cases,  the  blocking  variable  was  insignificant.  Furthermore,  the  results 
in  all  cases  were  consistent  with  those  achieved  using  absolute  error  as  the  response, 
showing  factors  C  and  H  as  significant. 

Summary 

The  statistical  analysis  tests  and  the  normal  probability,  Pareto,  and  Bayes  plots 
support  the  hypothesis  that  factors  C  and  H  are  significant.  Subsequent  analysis  indicates 
that  the  C*H  interaction  effect  is  not  significant.  The  blocking  effect  between  replications 
was  insignificant,  and  using  error  and  squared  error  as  response  variables  resulted  in 
identical  conclusions  to  those  achieved  using  the  absolute  error  response.  Analysis  of  the 
resulting  two-factor  model  reveals  that  availability  error  is  reduced  when  operating  at  a 
high  level  for  factor  C  (number  of  data  points)  and  a  low  level  for  factor  H  (system  size). 
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VII.  SUMMARY  AND  CONCLUSIONS 


Research  Objectives 

The  general  purpose  of  this  study  was  to  provide  insight  into  the  input  data 
characterization  factors  which  may  affect  the  accuracy  of  availability  model  output.  The 
potential  benefits  of  identifying  the  key  factors  would  be  the  reduction  of  unproductive 
data  collection  and  more  efficient  RM&A  modeling. 

.  The  overall  research  objectives  were  to: 

(1)  Identify  potential  factors  which  affect  availability  model  output  accuracy. 

(2)  Screen  the  potential  factors  to  determine  which  have  a  statistically  significant 

effect  (or  interaction  effect)  on  output  accuracy. 

(3)  Assess  the  magnitude  of  the  significant  effects. 

(4)  Provide  basic  insights  to  aid  in  efficient  component  input  data  characterization 

for  availability  models. 

Overview  of  Results 

Component  input  data  characterization  factors  thought  to  possibly  affect  system 
availability  estimates  were  identified  and  analyzed.  Using  a  design  of  experiment  approach 
with  the  absolute  error  of  system  availability  estimates  serving  as  the  response,  a  two- 
stage  experimental  screening  process  was  conducted  to  identify  the  active  factors. 

Preliminary  Experiment.  The  results  from  the  preliminary  experiment  were 
inconclusive,  identifying  number  of  data  points  and  fitting  method  for  the  important 
components  as  possible  significant  factors.  Using  absolute  error  as  the  response,  all 
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factors  proved  insignificant.  The  average  system  availability  estimate  absolute  error  was 
.3785%. 


Final  Experiment.  The  final  experiment,  analyzing  four  basic  structures,  revealed  that 
system  size  (5-component  versus  20-component)  and  the  number  of  data  points 
(5  versus  25)  do  affect  estimate  accuracy.  It  also  showed  that  fitting  technique,  underlying 
component  failure  distribution  (IFR  versus  DFR),  and  system  structure  type  (series- 
parallel  versus  complex)  do  not  have  a  significant  effect.  The  interaction  effect  between 
the  two  active  factors  was  not  statistically  significant.  Using  error  and  squared  error  as 
response  variables  resulted  in  the  same  conclusions  achieved  using  the  absolute  error 
response.  The  average  system  availability  estimate  absolute  error  was  3.4997%,  and  the 
effect  estimates  were  -3.504%  for  the  ‘number  of  data  points’  factor  and  3.1045%  for  the 
‘system  size’  factor.  The  response  surface  from  the  two- factor  model  derived  from  the 
final  experiment  showed  that  estimation  error  is  ntinimized  when  the  number  of  data 
points  is  at  a  high  level  and  the  system  size  is  small. 

Multivariate  Analysis.  The  supplemental  multivariate  analysis  of  RAPTOR  output 
(Appendix  G)  revealed  that  multivariate  techniques  can  be  used  to  discriminate  between 
various  structures  based  on  model  outputs.  It  was  also  discovered  that  structures  with 
predominantly  DFR  components  produce  higher  variability  in  RAPTOR  output  measures 
than  structures  with  predominantly  IFR  components. 
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Conclusions 


Several  insights  were  gained  from  this  research; 

(1)  More  availability  estimation  error  is  to  be  expected  when  analyzing  larger 
system  structures; 

(2)  Availability  estimation  error  can  be  reduced  by  increasing  the  number  of  failure 
and  repair  data  points  collected  for  each  system  component; 

(3)  There  is  no  measurable  significant  difference  in  estimation  error  when  analyzing 
systems  with  IFR  component  failure  characteristics  versus  systems  with  DFR 
component  failure  characteristics; 

(4)  There  is  no  apparent  benefit  in  focusing  on  ‘important’  versus  ‘non-important’ 
components  when  characterizing  component  failure  and  repair  probability 
distributions; 

(5)  There  is  no  apparent  difference  in  estimation  error  when  analyzing 
series-parallel  stmctures  versus  complex  structures;  and 

(6)  No  single  fitting  technique  utilized  in  this  research  provided  any  distinct 
advantage  over  any  other  method  for  availability  estimate  error  reduction. 

To  summarize,  the  availability  measure  appears  to  be  robust  to  fitting  method,  component 

failure  characteristics,  and  system  structure  type,  and  sensitive  to  the  number  of  data 

points  used  in  data  fitting  and  the  system  size. 


Comparison  with  Past  Research  Results 

Sensitivity  to  Component  Failure  Rate  Characterization.  In  analyzing  a  large 
space  system.  Wolf  found  very  tittle  sensitivity  of  the  predicted  system  availability  to 
individual  component  failure  rate  estimates  [23:69].  The  preliminary  experimental  results 
showed  that  the  number  of  data  points  may  affect  availability  estimation  accuracy.  The 
final  experiment  showed  conclusively  that  the  number  of  data  points  used  in  the 


60 


characterization  of  component  failure  and  repair  behavior  can  have  a  statistically 
significant  affect  on  availability  estimation  accuracy. 

Edgar  and  Bendell  concluded  that  failure  distributions  were  more  critical  than  repair 
distributions  in  defining  overall  system  behavior  and  that  decreasing  failure  rates  (DFR) 
were  more  critical  than  increasing  failure  rates  (IFR)  [24:125].  This  study  revealed  that, 
at  least  when  measuring  system  availability  estimation  error,  the  fitting  fidelity  of  the 
failure  and  repair  distributions  and  the  underlying  component  failure  rate  (IFR  versus 
DFR)  were  not  significant.  System  availability  appears  to  be  a  highly  robust  system 
characteristic  and  may  be  less  sensitive  than  other  system  characteristics  to  changes  in 
certain  factors.  The  multivariate  study  showed  that  DFR  component  structures  have 
higher  output  variability  than  IFR  component  stmctures. 

Exponential  Assumption.  Mortin,  Krolewski,  and  Cushing  provided  examples  where 
the  indiscriminate  use  of  the  exponential  distribution  for  component  failure 
characterization  can  produce  erroneous  results  [26:54].  In  this  study,  the  use  of  the 
exponential  distribution  for  component  failure  characterization  (when  the  true  underlying 
failure  distribution  was  Weibull)  did  not  significantly  alter  system  availability  estimation 
accuracy.  Again,  this  may  indicate  that  the  availability  measure  is  robust  to  component 
distributional  assumptions. 

Suggestions  for  Further  Research 

Identifying  Other  Factors.  The  final  regression  model  (using  the  absolute  error 
response)  explained  only  a  portion  of  the  overall  response  measure  variability  with  an  R^ 
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of  .297,  suggesting  that  other  significant  explanatory  variables  may  exist.  More  formal 
methods  could  be  conducted  to  identify  other  possible  critical  input  data  characterization 
factors  not  addressed  in  this  study,  such  as  a  formal  survey  of  several  Air  Force  reliability 
analysts.  A  screening  design  could  then  be  accomplished  to  identify  other  significant 
factors. 

Mean-Time-to- Failure  /  Mean-Repair-Time  (MTTF/MRT)  Ratio.  After  reviewing 
the  results  of  the  preliminary  experiment,  AFOTEC  analysts  felt  one  important  factor  to 
analyze  would  be  the  component  MTTF/MRT  ratio.  They  suspected  that  system 
availability  estimates  might  be  more  sensitive  to  some  of  the  factors  analyzed  in  this  study 
when  several  components  possessed  a  low  MTTF/MRT  ratio.  Time  did  not  allow  for  the 
inclusion  of  the  MTTF/MRT  factor  in  this  study;  in  fact,  it  was  randomized  in  the 
experimental  design  to  mitigate  (‘spread  around’)  its  effect.  Follow-on  experiments  which 
incorporate  this  factor  may  be  insightful. 

Response  Surface  Methodology  (RSM).  This  research  addressed  qualitative  as  well 
as  quantitative  factors.  In  all  cases,  the  qualitative  factors  proved  insignificant.  However, 
two  quantitative  factors  (number  of  data  points  and  system  size)  were  significant.  A 
simple  linear  response  surface  was  developed  for  the  resultant  two-factor  model  for  the 
defined  experimental  region.  The  factor  levels  used  for  the  experiment  (number  of  data 
points:  5  and  25;  system  size:  5  components  and  20  components)  represents  a  limited 
experimental  region.  Using  RSM,  the  experimental  region  could  be  expanded  and 
explored  in  more  detail. 
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Appendix  A:  Statistical  Analysis  Output 


Preliminary  Experiment:  JMP  Output  (Without  Blocking  Variable) 


Screening  Fit 
ABS  Error 
Summary  of  Fit 


RSquare 

0.004537 

RSquare  Adj 

-0.11397 

Root  Mean  Square  Error 

0.30666 

Mean  of  Response 

0.378498 

Observations  (or  Sum  Wgts) 

48 

Analysis  of  Variance 

Source 

DF  Sum  of  Squares  Mean  Square 

F  Ratio 

Model 

5  0.0180009 

0.003600 

0.0383 

Error 

42  3.9496998 

0.094040 

Prob>F 

C  Total 

47  3.9677006 

0.9991 

Lack  of  Fit 

Source 

DF  Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  of  Fit 

10  0.0758773 

0.007588 

0.0627 

Pure  Error 

32  3.8738225 

0.121057 

Prob>F 

Total  Error 

42  3.9496998 

1.0000 

Max  RSq 

0.0237 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

Intercept  0.3784979  0.044263 

8.55 

<.0001 

A 

-0.001931  0.044263 

-0.04 

0.9654 

B 

-0.013469  0.044263 

-0.30 

0.7624 

C 

0.0096646  0.044263 

0.22 

0.8282 

D 

-0.009356  0.044263 

-0.21 

0.8336 

E 

0.0029896  0.044263 

0.07 

0.9465 

Effect  Test 

Source 

Nparm  DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1  1  0.00017903 

0.0019 

0.9654 

B 

1  1  0.00870755 

0.0926 

0.7624 

C 

1  1  0.00448340 

0.0477 

0.8282 

D 

1  1  0.00420189 

0.0447 

0.8336 

E 

1  1  0.00042901 

0.0046 

0.9465 

Error 

Summary  of  Fit 

RSquare 

0.057138 

RSquare  Adj 

-0.05511 

Root  Mean  Square  Error 

0.427632 

Mean  of  Response 

0.237094 

Observations  (or  Sum  Wgts) 

48 
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Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

5 

0.4654480 

0.093090 

0.5090 

Error 

42 

7.6805195 

0.182870 

Prob>F 

C  Total 

47 

8.1459675 

0.7678 

Lack  of  Fit 


Source 

DF  Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  of  Fit 

10  0.0345872 

0.003459 

0.0145 

Pure  Error 

32  7.6459323 

0.238935 

Prob>F 

Total  Error 

42  7.6805195 

1.0000 

Max  RSq 
0.0614 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.2370937  0.061723 

3.84 

0.0004 

A 

0.0841313  0.061723 

1.36 

0.1801 

B 

-0.000431  0.061723 

-0.01 

0.9945 

C 

0.0507771  0.061723 

0.82 

0.4154 

D 

-0.003435  0.061723 

-0.06 

0.9559 

E 

0.0053354  0.061723 

0.09 

0.9315 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1  0.33974723 

1.8579 

0.1801 

B 

1 

1  0.00000893 

0.0000 

0.9945 

C 

1 

1  0.12375899 

0.6768 

0.4154 

D 

1 

1  0.00056650 

0.0031 

0.9559 

E 

1 

1  0.00136640 

0.0075 

0.9315 

SQ  Error 

Summary  of  Fit 

RSquare 

0.020457 

RSquare  Adj 

-0.09616 

Root  Mean  Square  Error 

0.274629 

Mean  of  Response 

0.225921 

Observations  (or  Sum  Wgts) 

48 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

5 

0.0661547 

0.013231 

0.1754 

Error 

42 

3.1676947 

0.075421 

Prob>F 

C  Total 

47 

3.2338494 

0.9703 

Lack  of  Fit 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  of  Fit 

10 

0.0548711 

0.005487 

0.0564 

Pure  Error 

32 

3.1128236 

0.097276 

Prob>F 

Total  Error 

42 

3.1676947 

1.0000 

Max  RSq 
0.0374 


64 


Parameter  Estimates 


Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.2259211  0.039639 

5.70 

<.0001 

A 

-0.01106  0.039639 

-0.28 

0.7816 

B 

-0.018303  0.039639 

-0.46 

0.6466 

C 

0.0275692  0.039639 

0.70 

0.4906 

D 

-0.011699  0.039639 

-0.30 

0.7693 

E 

0.0048954  0.039639 

0.12 

0.9023 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1  0.00587120 

0.0778 

0.7816 

B 

1 

1  0.01608043 

0.2132 

0.6466 

C 

1 

1  0.03648302 

0.4837 

0.4906 

D 

1 

1  0.00656969 

0.0871 

0.7693 

E 

1 

1  0.00115031 

0.0153 

0.9023 

Preliminary  Experiment:  JMP  Output  (With  Blocking  Variable) 

Screening  Fit 
ABS  Error 
Summary  of  Fit 


RSquare  0.526943 

RSquare  Adj  0.444158 

Root  Mean  Square  Error  0.216619 

Mean  of  Response  0.37 8498 

Observations  (or  Sum  Wgts)  48 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares  Mean  Square 

F  Ratio 

Model 

7 

2.0907506 

0.298679 

6.3652 

Error 

40 

1.8769500 

0.046924 

Prob>F 

C  Total 

47 

3.9677006 

<.0001 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.3784979 

0.031266 

12.11 

<.0001 

A 

-0.001931 

0.031266 

-0.06 

0.9511 

B 

-0.013469 

0.031266 

-0.43 

0.6689 

C 

0.0096646 

0.031266 

0.31 

0.7588 

D 

-0.009356 

0.031266 

-0.30 

0.7663 

E 

0.0029896 

0.031266 

0.10 

0.9243 

BIock[l-3] 

-0.291992 

0.044217 

-6.60 

<.0001 

Block[2-3] 

0.1172021 

0.044217 

2.65 

0.0115 
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Effect  Test 


Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1  0.0001790 

0.0038 

0.9511 

B 

1 

1  0.0087075 

0.1856 

0.6689 

C 

1 

1  0.0044834 

0.0955 

0.7588 

D 

1 

1  0.0042019 

0.0895 

0.7663 

E 

1 

1  0.0004290 

0.0091 

0.9243 

Block 

2 

2  2.0727498 

22.0864 

<.0001 

Error 

Summary  of  Fit 

RSquare 

0.434239 

RSquare  Adj 

0.335231 

Root  Mean  Square  Error 

0.339436 

Mean  of  Response 

0.237094 

Observations  (or  Sum  Wgts) 

48 

Analysis  of  Variance 

Source 

DF 

Sum  of  Squares  Mean  Square 

F  Ratio 

Model 

7 

3.5372968 

0.505328 

4.3859 

Error 

40 

4.6086707 

0.115217 

Prob>F 

C  Total 

47 

8.1459675 

0.0011 

Parameter  Estimates 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.2370937  0.048993 

4.84 

<.0001 

A 

0.0841313  0.048993 

1.72 

0.0937 

B 

-0.000431  0.048993 

-0.01 

0.9930 

C 

0.0507771  0.048993 

1.04 

0.3062 

D 

-0.003435  0.048993 

-0.07 

0.9444 

E 

0.0053354  0.048993 

0.11 

0.9138 

Block[l-3] 

-0.30305  0.069287 

-4.37 

<.0001 

Block[2-3] 

-0.013144  0.069287 

-0.19 

0.8505 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1  0.3397472 

2.9488 

0.0937 

B 

1 

1  0.0000089 

0.0001 

0.9930 

C 

1 

1  0.1237590 

1.0741 

0.3062 

D 

1 

1  0.0005665 

0.0049 

0.9444 

E 

1 

1  0.0013664 

0.0119 

0.9138 

Block 

2 

2  3.0718488 

13.3307 

<.0001 

SQ  Error 

Summary  of  Fit 

RSquare 

0.373498 

RSquare  Adj 

0.26386 

Root  Mean  Square  Error 

0.225056 

Mean  of  Response 

0.225921 

Observations  (or  Sum  Wgts) 

48 
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Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

7 

1.2078352 

0.172548 

3.4066 

Error 

40 

2.0260142 

0.050650 

Prob>F 

C  Total 

47 

3.2338494 

0.0060 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.2259211 

0.032484 

6.95 

<.0001 

A 

-0.01106 

0.032484 

-0.34 

0.7353 

B 

-0.018303 

0.032484 

-0.56 

0.5763 

C 

0.0275692 

0.032484 

0.85 

0.4011 

D 

-0.011699 

0.032484 

-0.36 

0.7206 

E 

0.0048954 

0.032484 

0.15 

0.8810 

Block!  1-3] 

-0.215254 

0.045939 

-4.69 

<.0001 

Block[2-3] 

0.0771839 

0.045939 

1.68 

0.1007 

Effect  Test 

Source  Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

0.0058712 

0.1159 

0.7353 

B 

1 

1 

0.0160804 

0.3175 

0.5763 

C 

1 

1 

0.0364830 

0.7203 

0.4011 

D 

1 

1 

0.0065697 

0.1297 

0.7206 

E 

1 

1 

0.0011503 

0.0227 

0.8810 

Block 

2 

2 

1.1416805 

11.2702 

0.0001 

Final  Experiment  -  Full  Main  Effect  Model:  JMP  Output  (Without  Blocking  Variable) 

Screening  Fit 
Abs  Error 
Summary  of  Fit 

RSquare  0.333092 

RSquareAdj  0.102239 

Root  Mean  Square  Error  4.64597 1 


Mean  of  Response 

3.499722 

Observations  (or  Sum  Wgts) 

36 

Analysis  of  Variance 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

9 

280.30078 

31.1445 

1.4429 

Error 

26 

561.21112 

21.5850 

Prob>F 

C  Total 

35 

841.51190 

0.2214 

Lack  of  Fit 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  of  Fit 

2 

1.51785 

0.7589 

0.0325 

Pure  Error 

24 

559.69327 

23.3206 

Prob>F 

Total  Error 

26 

561.21112 

0.9680 

Max  RSq 
0.3349 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

3.4997222 

0.774328 

4.52 

0.0001 

A 

0.4468611 

0.774328 

0.58 

0.5688 

B 

-0.916772 

0.774328 

-1.18 

0.2471 

C 

-1.770133 

0.774328 

-2.29 

0.0306 

D 

-0.021161 

0.774328 

-0.03 

0.9784 

E 

0.3164833 

0.774328 

0.41 

0.6861 

F 

-0.269144 

0.774328 

-0.35 

0.7310 

G 

0.6426 

0.774328 

0.83 

0.4142 

H 

1.5522389 

0.774328 

2.00 

0.0555 

I 

-0.785606 

0.774328 

-1.01 

0.3197 

Effect  Test 

Source 

Nparm  DF  Sum 

of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

7.18865 

0.3330 

0.5688 

B 

1 

1 

30.25697 

1.4018 

0.2471 

C 

1 

1 

112.80139 

5.2259 

0.0306 

D 

1 

1 

0.01612 

0.0007 

0.9784 

E 

1 

1 

3.60582 

0.1671 

0.6861 

F 

1 

1 

2.60779 

0.1208 

0.7310 

G 

1 

1 

14.86565 

0.6887 

0.4142 

H 

1 

1 

86.74004 

4.0185 

0.0555 

I 

1 

1 

22.21834 

1.0293 

0.3197 

Error 

Summary  of  Fit 

RSquare 

0.364237 

RSquare  Adj 

0.144165 

Root  Mean  Square  Error 

5.134356 

Mean  of  Response 

2.3826 

Observations  (or  Sum  Wgts) 

36 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

9 

392.6757 

43.6306 

1.6551 

Error 

26 

685.4020 

26.3616 

Prob>F 

C  Total 

35 

1078.0777 

0.1515 

Lack  of  Fit 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  of  Fit 

2 

7.62238 

3.8112 

0.1350 

Pure  Error 

24 

677.77964 

28.2408 

Prob>F 

Total  Error 

26 

685.40201 

0.8744 

Max  RSq 
0.3713 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>lt| 

Intercept 

2.3826 

0.855726 

2.78 

0.0099 

A 

0.9243389 

0.855726 

1.08 

0.2900 

B 

-0.987061 

0.855726 

-1.15 

0.2592 

C 

-1.6128 

0.855726 

-1.88 

0.0707 

D 

-0.121572 

0.855726 

-0.14 

0.8881 

E 

-0.203883 

0.855726 

-0.24 

0.8135 

F 

-0.784844 

0.855726 

-0.92 

0.3675 

G 

0.8526778 

0.855726 

1.00 

0.3282 

H 

2.2535167 

0.855726 

2.63 

0.0140 

I 

0.0108167 

0.855726 

0.01 

0.9900 

Effect  Test 

Source 

Nparm  DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

30.75849 

1.1668 

0.2900 

B 

1 

1 

35.07443 

1.3305 

0.2592 

C 

1 

1 

93.64046 

3.5522 

0.0707 

D 

1 

1 

0.53207 

0.0202 

0.8881 

E 

1 

1 

1.49646 

0.0568 

0.8135 

F 

1 

1 

22.17531 

0.8412 

0.3675 

G 

1 

1 

26.17414 

0.9929 

0.3282 

H 

1 

1 

182.82015 

6.9351 

0.0140 

I 

1 

1 

0.00421 

0.0002 

0.9900 

SQ  Error 

Summary  of  Fit 

RSquare 

0.295785 

RSquare  Adj 

0.052019 

Root  Mean  Square  Error 

97.38942 

Mean  of  Response 

35.62339 

Observations  (or  Sum  Wgts) 

36 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

9 

103578.18 

11508.7 

1.2134 

Error 

26 

246602.20 

9484.7 

Prob>F 

C  Total 

35 

350180.38 

0.3290 

Lack  of  Fit 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  of  Fit 

2 

922.26 

461.1 

0.0450 

Pure  Error 

24 

245679.94 

10236.7 

Prob>F 

Total  Error 

26 

246602.20 

0.9560 

Max  RSq 
0.2984 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

35.623386 

16.23157 

2.19 

0.0373 

A 

17.173384 

16.23157 

1.06 

0.2998 

B 

-22.02642 

16.23157 

-1.36 

0.1864 

C 

-30.17743 

16.23157 

-1.86 

0.0744 

D 

1.7020264 

16.23157 

0.10 

0.9173 

E 

-0.217741 

16.23157 

-0.01 

0.9894 

F 

-2.477217 

16.23157 

-0.15 

0.8799 

G 

8.4618496 

16.23157 

0.52 

0.6066 

H 

27.486565 

16.23157 

1.69 

0.1023 

I 

-18.71392 

16.23157 

-1.15 

0.2594 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

10617.304 

1.1194 

0.2998 

B 

1 

1 

17465.871 

1.8415 

0.1864 

C 

1 

1 

32784.393 

3.4566 

0.0744 

D 

1 

1 

104.288 

0.0110 

0.9173 

E 

1 

1 

1.707 

0.0002 

0.9894 

F 

1 

1 

220.918 

0.0233 

0.8799 

G 

1 

1 

2577.704 

0.2718 

0.6066 

H 

1 

1 

27198.406 

2.8676 

0.1023 

I 

1 

1 

12607.593 

1.3293 

0.2594 

Final  Experiment  -  Full  Main  Effect  Model:  JMP  Output  (With  Blocking  Variable' 

Screening  Fit 
Abs  Error 
Summary  of  Fit 

RSquare  0.369889 

RSquareAdj  0.081088 

Root  Mean  Square  Error  4.70038 

Mean  of  Response  3 .499722 

Observations  (or  Sum  Wgts)  36 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

11 

311.26611 

28.2969 

1.2808 

Error 

24 

530.24579 

22.0936 

Prob>F 

C  Total 

35 

841.51190 

0.2931 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

3.4997222 

0.783397 

4.47 

0.0002 

A 

0.4468611 

0.783397 

0.57 

0.5737 

B 

-0.916772 

0.783397 

-1.17 

0.2534 

C 

-1.770133 

0.783397 

-2.26 

0.0332 

D 

-0.021161 

0.783397 

-0.03 

0.9787 

E 

0.3164833 

0.783397 

0.40 

0.6898 

F 

-0.269144 

0.783397 

-0.34 

0.7342 

G 

0.6426 

0.783397 

0.82 

0.4201 

H 

1.5522389 

0.783397 

1.98 

0.0591 

I 

-0.785606 

0.783397 

-1.00 

0.3260 

Block[l-3] 

1.2199611 

1.10789 

1.10 

0.2817 

Block[2-3] 

-1.027106 

1.10789 

-0.93 

0.3631 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

7.18865 

0.3254 

0.5737 

B 

1 

1 

30.25697 

1.3695 

0.2534 

C 

1 

1 

112.80139 

5.1056 

0.0332 

D 

1 

1 

0.01612 

0.0007 

0.9787 

E 

1 

1 

3.60582 

0.1632 

0.6898 

F 

1 

1 

2.60779 

0.1180 

0.7342 

G 

1 

1 

14.86565 

0.6728 

0.4201 

H 

1 

1 

86.74004 

3.9260 

0.0591 

I 

1 

1 

22.21834 

1.0056 

0.3260 

Block 

2 

2 

30.96533 

0.7008 

0.5061 

Error 

Summary  of  Fit 

RSquare 

0.415505 

RSquare  Adj 

0.147612 

Root  Mean  Square  Error 

5.124008 

Mean  of  Response 

2.3826 

Observations  (or  Sum  Wgts) 

36 

Analysis  of  Variance 

Source 

DF 

Sum  of  Squares  Mean  Square 

F  Ratio 

Model 

11 

447.9467 

40.7224 

1.5510 

Error 

24 

630.1310 

26.2555 

Prob>F 

C  Total 

35 

1078.0777 

0.1778 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

2.3826 

0.854001 

2.79 

0.0102 

A 

0.9243389 

0.854001 

1.08 

0.2898 

B 

-0.987061 

0.854001 

-1.16 

0.2591 

C 

-1.6128 

0.854001 

-1.89 

0.0711 

D 

-0.121572 

0.854001 

-0.14 

0.8880 

E 

-0.203883 

0.854001 

-0.24 

0.8133 

F 

-0.784844 

0.854001 

-0.92 

0.3672 

G 

0.8526778 

0.854001 

1.00 

0.3280 

H 

2.2535167 

0.854001 

2.64 

0.0144 

I 

0.0108167 

0.854001 

0.01 

0.9900 

Block[l-3] 

1.45715 

1.20774 

1.21 

0.2394 

Block[2-3] 

-1.571483 

1.20774 

-1.30 

0.2056 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

30.75849 

1.1715 

0.2898 

B 

1 

1 

35.07443 

1.3359 

0.2591 

C 

1 

1 

93.64046 

3.5665 

0.0711 

D 

1 

1 

0.53207 

0.0203 

0.8880 

E 

1 

1 

1.49646 

0.0570 

0.8133 

F 

1 

1 

22.17531 

0.8446 

0.3672 

G 

1 

1 

26.17414 

0.9969 

0.3280 

H 

1 

1 

182.82015 

6.9631 

0.0144 

I 

1 

1 

0.00421 

0.0002 

0.9900 

Block 

2 

2 

55.27102 

1.0526 

0.3646 

SQ  Error 

Summary  of  Fit 

RSquare 

0.374379 

RSquare  Adj 

0.087636 

Root  Mean  Square  Error 

95.54236 

Mean  of  Response 

35.62339 

Observations  (or  Sum  Wgts) 

36 

Analysis  of  Variance 

Source 

DF 

Sum  of  Squares  Mean  Square 

F  Ratio 

Model 

11 

131100.15 

11918.2 

1.3056 

Error 

24 

219080.23 

9128.3 

Prob>F 

C  Total 

35 

350180.38 

0.2802 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

35.623386 

15.92373 

2.24 

0.0348 

A 

17.173384 

15.92373 

1.08 

0.2915 

B 

-22.02642 

15.92373 

-1.38 

0.1793 

C 

-30.17743 

15.92373 

-1.90 

0.0702 

D 

1.7020264 

15.92373 

0.11 

0.9158 

E 

-0.217741 

15.92373 

-0.01 

0.9892 

F 

-2.477217 

15.92373 

-0.16 

0.8777 

G 

8.4618496 

15.92373 

0.53 

0.6000 

H 

27.486565 

15.92373 

1.73 

0.0972 

I 

-18.71392 

15.92373 

-1.18 

0.2514 

Block[l-3] 

38.210871 

22.51955 

1.70 

0.1027 

Block[2-3] 

-26.2954 

22.51955 

-1.17 

0.2544 

Effect  Test 

Source 

Nparm 

DF  Sum  of  Squares 

F  Ratio 

Prob>F 

A 

1 

1 

10617.304 

1.1631 

0.2915 

B 

1 

1 

17465.871 

1.9134 

0.1793 

C 

1 

1 

32784.393 

3.5915 

0.0702 

D 

1 

1 

104.288 

0.0114 

0.9158 

E 

1 

1 

1.707 

0.0002 

0.9892 

F 

1 

1 

220.918 

0.0242 

0.8777 

G 

1 

1 

2577.704 

0.2824 

0.6000 

H 

1 

1 

27198.406 

2.9796 

0.0972 

I 

1 

1 

12607.593 

1.3811 

0.2514 

Block 

2 

2 

27521.966 

1.5075 

0.2417 

Final  Exoeiiment 

-C.H-C*H  Model;  JMPOutnut 

Response  Variable:  Absolute  Error 

Screening  Fit 


Source 

Y 

Summary  of  Fit 

RSquare 

RSquare  Adj 

Root  Mean  Square  Error 

Mean  of  Response 

Observations  (or  Sum  Wgts) 

Analysis  of  Variance 

DF  Sum  of  Squares 

0.296981 

0.231073 

4.299705 

3.499722 

36 

Mean  Square 

F  Ratio 

Model 

3 

249.91310 

83.3044 

4.5060 

Error 

32 

591.59880 

18.4875 

Prob>F 

C  Total 

35 

841.51190 

0.0095 
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X  o 


Parameter  Estimates 


C*H 


Term 

Estimate 

Std  Error 

t  Ratio 

Intercept 

3.4997222 

0.716617 

4.88 

C*H 

-1.182883 

0.716617 

-1.65 

C 

-1.770133 

0.716617 

-2.47 

H 

1.5522389 

0.716617 

2.17 

Effect  Test 

e  Nparm 

DF  Sum  of  Squares 

F  Ratio 

1 

1 

50.37167 

2.7246 

1 

1 

112.80139 

6.1015 

1 

1 

86.74004 

4.6918 

Prediction  Profile 

C 


H 


Prob>|t| 

<.0001 

0.1086 

0.0190 

0.0379 


Prob>F 

0.1086 

0.0190 

0.0379 
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Appendix  B:  Final  Experiment  Structures 
and  True  Component  Distribution  Functions 


Small  /  Series-Parallel: 


Large  /  Series-Parallel: 


h 


LleP 


rl_Ui 


4lEP 


Li 


10 


H 


rODn 

h-c 


LgEH 


rDEh 

-DD- 


18 


Lrir~|-i 


Large  /  Complex: 
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Component  True  Failure  and  Repair  Distributions  (Final  Experiment) 


14/15/16 


18/19/20 


IFR 

Failure  Distribution 

DFR 

Failure  Distribution 

Repair 

Distribution 

Weibull:  Shape  =1.5  (hrs) 
Scale  =  3000 
Location  =  0 

Weibull:  Shape  =  .50  (hrs) 
Scale  =  1354 
Location  =  0 

Lognormal:  (hrs) 

Mean=  2800 

S.D.  =  200 

Weibull:  Shape  =  4.0 

Scale  =  2500 
Location  =  0 

Weibull:  Shape  =  .85 

Scale  =  2082 
Location  =  0 

Lognormal: 

Mean=  1500 

S.D.  =  100 

Weibull:  Shape  =  2.5 

Scale  =  4000 
Location  =  0 

Weibull:  Shape  =  .95 

Scale  =  3468 
Location  =  0 

Lognormal: 

Mean=  1000 

S.D.  =  150 

Weibull:  Shape  =  1.7 

Scale  =  1700 
Location  =  0 

Weibull:  Shape  =  .60 

Scale  =  1008 
Location  =  0 

Lognormal: 

Mean=  150 

S.D.  =  25 

Weibull:  Shape  =  2.8 

Scale  =  3500 
Location  =  0 

Weibull:  Shape  =  .40 

Scale  =  938 

Location  =  0 

Lognormal: 

Mean  =  850 

S.D.  =  90 

Weibull:  Shape  =  1.9 

Scale  =  3333 
Location  =  0 

Weibull:  Shape  =  .70 

Scale  =  2336 
Location  =  0 

Lognormal: 

Mean=  3000 

S.D.  =  125 

Weibull:  Shape  =  1.2 

Scale  =  2575 
Location  =  0 

Weibull:  Shape  =  .55 

Scale  =  1423 
Location  =  0 

Lognormal: 

Mean  =  190 

S.D.  =  20 

Weibull:  Shape  =  2.7 

Scale  =  1500 
Location  =  0 

Weibull:  Shape  =  .78 

Scale  =  1 156 
Location  =  0 

Lognormal: 

Mean=  1200 

S.D.  =  75 

Weibull:  Shape  =  1.6 

Scale  =  6000 
Location  =  0 

Weibull:  Shape  =  .91 

Scale  =  5143 
Location  =  0 

Lognormal: 

Mean=  1000 

S.D.  =  30 

Weibull:  Shape  =  2.3 

Scale  =  4700 
Location  =  0 

Weibull:  Shape  =  .46 

Scale  =  1763 
Location  =  0 

Lognormal: 

Mean=  2300 

S.D.  =  133 

Weibull:  Shape  =1.4 

Scale  =  2700 
Location  =  0 

Weibull:  Shape  =  .82 

Scale  =  2210 
Location  =  0 

Lognormal: 

Mean  =  500 

S.D.  =  60 

Weibull:  Shape  =  1.9 

Scale  =  2700 
Location  =  0 

Weibull:  Shape  =  .67 

Scale  =  1812 
Location  =  0 

Lognormal: 

Mean=  1000 

S.D.  =  100 

Weibull:  Shape  =  1.3 

Scale  =  4200 
Location  =  0 

Weibull:  Shape  =  .86 

Scale  =  3591 
Location  =  0 

Lognormal: 

Mean=  90 

S.D.  =  15 

Weibull:  Shape  =  1.5 

Scale  =  2600 
Location  =  0 

Weibull:  Shape  =  .62 

Scale  =  1626 
Location  =  0 

Lognormal: 

Mean=  2200 

S.D.  =  200 

Weibull:  Shape  =  1.1 

Scale  =  3100 
Location  =  0 

Weibull:  Shape  =  .75 

Scale  =  2513 
Location  =  0 

Lognormal: 

Mean  =  750 

S.D.  =  60 

Weibull:  Shape  =  1.6 

Scale  =  2000 
Location  =  0 

Weibull:  Shape  =  .48 

Scale  =  829 

Location  =  0 

Lognormal: 

Mean=  280 

S.D.  =  50 
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Appendix  C:  Fitting  Data  (Preliminary  Experiment) 
Components  1  and  2: 


78 


Failure  Oat 

a 

Sets 

2207.199 

1785.776 

Shape 

Scale 

2657.587 

Location 

5230.137 

1682.83 

Nornial 

1131.662 

Mean 

2304.446 

S.D. 

2220.419 

818.5274 

459.4963 

Sets 

1696.307 

1365.369 

Shape 

Scale 

2687.736 

Location 

1805.908 

2860.507 

2341 .057 

2884.308 

240.2723 

4878.578 

2950.525 

4238.523 

1467.667 

1540.711 

(Top  Weibull++  Selection) _ 

High  Level  Fitting  Patamelers 
Rep1  Rep2  Rep3 

_ 1.842  1.701 

_ 1808.529  2257.799 

317.4287  43.7521 


(Top  Weibull++  Selection) 
High  Level  Fitting  Parameters 


Repi 

1.2122 


Rep2 

2.0219 


Rep3 

1.6765 


1663.22  2204.209  2125.105 


(Weibull++  Exponential) 

Low  Level  Fitting  Parameters 


1250 

1250 

1666.667 

14.1518 

664.5628 

459.4963 

(Weibull++ 

Exponential)  ' 

Low  Level  Fitting  Parameters 

iKkEUh 


3350.027  649.1847 


641.14391  4166.519  1783.356 
814.9092  2043.477  80.483 


817.0835 

4346.261 

817.7518 

1992.607 

2434.073 

1013.223 

364.0332 

701.3264 

1415.027 

1327.958 

1119,601 

4072.3241 

1162.011 

'MSE^ 

1323.632 

iKHWWil 

■  FlMKHri 

mUfUeKK 

■RggRgg 

iilritjclUm 

krirfitkl:! 

■BltiliBM 

\VtlA\ 

KiVm-e! 

■l.-W*bk 

KiMA' 

ETtgliS 

■c»i!;ltkZ! 

IllllBBi 

■EH 

■MUiSljK 

tBiiaiiat 

BftTSgril' 

m^um 

Wdt'MVit 

mmmi 

mm*>i 

■.-liHIiMt: 
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80 


lm.Jlll.l.llMlll.1 


Failure  Data 


lli.i!.-fi!i<ltlli:!{t;llt^yi 

IggERglMiSa^l 

iK^S^BBEESni 

lllc!{m.-flKlg?it;!cl^3cMI 

l■^t«^lMtyig£[llt1W^Mi 


244.2498 
473.7554  ~ 
774.6268  ~ 
1200.055  ~ 

1259.333  ^ 
1431.017" 
1463.062  ~ 
1628.846 ' 
2380.459 ' 

2878.333 


_ (Top  Weibull++  Selection) _ 

_ High  Level  Fitting  Patamelers 

_ Repi  Rep2  Rep3 

Shape  1.8723  1.1411  1.8549 

Scale  2014.397  1492.544  1545.62 

Location  0  530.6213  0 


_ (Weibull++  Exponential) _ 

_ Low  Level  Fitting  Parameters 

_ Repi  Rep2  Rep3 

0.0007  0.0006  0.0009 

_ 1428.571  1666.667  1111.111 

ocation  384.4927  440.0948  244.2498 


\mss^ 


atiMititiii 


|liM!laidlEgEaE!!E£! 


_ (Top  Weibull++  Selectbn) _ 

_ High  Level  Fitting  Parameters 

_ Rep1  Rep2  Rep3 

Shape  2.2201  1.7751  1.8758 

Scale  2154.565  1712.647  1853.406 
Location  0  0  0 


Lambda 

Mean 

Location 


(Weibull++  Exponential) _ 

Low  Level  Fitting  Parameters 
Repi  Rep2  Rep3 
0.0007  0.0007  0.0007 

1428.571  1428.571  1428.571 
478.862  139.3333  244.5103 


TRUE  PARAMETERS 
Weibull 

Shape  2 

Scale  2000 

Location  0 


1023.576  1040.816 


1407.178  1053.333  1086.567 


1424.359  1101.47  1120.212 


1452.061 


■t.iapricicj»KL.i.TggBnaH  , 

K^smw^mE^[am 

M  Itlil  Ktidteai  BiTM  I 

■  WtUMiMfctBPKia’i 

BEBmSB 


1967.877 
1994.657 
2029.51 
2038.27 
2122.888 
2235.849 
2375,611 
2421 .885 
2436.823 
2460.293 
2526.509 
2549.512 
2637.84 
2680.403 
2842.487 
2864.291 
2932.771 
3146.245 
3370.188 
3651.116 
3730.11 
4825.686 


■ci;iiiB:irilll<!it!.-^fey 

gM«aWiB1gSiMt:Kill 


81 


82 


TTOTTirtai 


a  I  Mg 


■BiTikW/ihlB-mtJ.-'ili 


I  HiB^^ 


|KnS^HEES^3IK£SSE]| 

lK£3i^iEE2H^KE££^Si 

l»:li'*!iiM»jiH>^»Ijjj«gli 

l»Kii!riai«!T5iiEr5n»y;-j*irf2Hi 

ywy*i:imfctiaEaiafcyi=i*-'i=:*ii 

lfcM-MMgHM?iEHEiElgliIiai3S[^i 

ItUKillliWMgniigHMiHiHBglili 

l■^i^2■]H:6MM^!^-!ie^gr<1!l^'2=i!!^■^^^:<lli 


ill.!.!=.IIJiMMiT:gMJ;l— n»IE1il^ 


Scale 

Location 

0 

Component  1:  DFR  Failure 


|lJ^IIHlL.ld»J3M 


1 1[||||||^^^^  j 


I 

l■E!i^j■^l^-!gH■=^^k^>-llld^^:Ba^|g:;^B^iflgEl^^^^!Bl!Ii^l 


I 

I 

inggTOW^TIMMMnMMP] 


TRUE  DFR  PARAMETER 
Welbull  T~ 

_ Shape _ I 

_ Scale  1354 

Location 


gjl^^l - 

■^feislsifcfell 

IdtahfcthiaijljjjjjkljlEjjjjjjEli 


im^liEiCliB 


■»;lli=l.|j;IJJ;M,^l 


ll^BBEiBal 


Sf:»'<li 


_ EESSB^EEiiEj^EESjE.- 


lM.-'j|cg!ciyilMBei!llH:HB!aMteli 


IW‘'trii'^l»'-'«l=l*‘«HB[iMII.-!tlfdi 

imuATtie»im-^!sitiWiim>jj']i\,*n 

l■^^■!;l!l^!H:l.^ggSfi^:r;^AlI»>iMt!M:^ili 

l^<m^.-^=v>J■.-'.;!.-gr»-^l»cMit=^[=Wil■ 

Ignn^iE^gtE^gGgaE^^ 

lESEEnOSSEESlKEBE^I 


5  Data  Points 


1367.8815 


Hioh  Level  Fittina  Parameters 


■icmt.;;;H»H?gET:!inBEPng^ 

■Icti'JMIMBIHiaiaSI^KgclKPiTa 


LogN  Mean 
LooNS.D. 


True  Lopnormal  Mean: 


True  Loanormat  St  Dev 


Low  Level  Fittina  Parameters 


ll13?H*roTniBBBnHflISW73?r!BF3ff?0»?gB^^T*T»!iTill 

IlgffirmB - 

IHH 

insiMBinigiH 


^HHHHiH[i^^HI!S!uSHI 


Mean  for  Normal  variates: 


varfor  Normal  variates; 


St  Dev  for  Normal  Variates; 


iKljl^l 


ll;fJlMlldti3l 

iBaiBBTHT 


iiidnramgin 


iii.ii,i|i:iijji'jiiii.i.i:;.i;--iMyijMi 


IMw;:iiigBiBlkr>-llky;EiB^5gl^MramEEini 


iMIc^cBHiigaElticfcH-MBIcHKKMI 


Ig|c?:!.-ilg:I;IBlBf3iyriBgrj5gE!M|j|llHIII.HiM.M^regCSlM 


iBH»a=K4iit!gtigfcai 

IlSgaHEIMHiitWiiHI 

mjsa^M#gfei!Hi 


Location 

lEldik^ 

1462.5032 

1469.2258 

1510.4431 

1511.885 

1525.1673 

1526.01 

1541 .3123 

1573.9059 

1595.3293 

1618.5044 

1631 .1695 

1641 .2894 

1678.8727 
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Component  3 


iraiismn 

■■m 

usnMSJsnMi^ 

jguriigiMtja 

^bihi 

■■■■■ 

(T  op  Weibull-i-+  Selection) 

tvaatiiM 

^EI3!3 

2.5 

Set1 

S©t2 

liilJSH 

ISE^H 

HEB 

Piitami 

■U!l!l>Ic 

■KEiM 

ItsInUUB 

Hll!l>i!li 

■bl!l!ll] 

H.!4.\IP.1iT 

^^HE) 

■t!ilt!!l!!:l:e! 

PiVtlikUi 

Scale 

■EHKHI 

■EE^ia^ 

EMAMgK 

■•J:m.-EE! 

0 

irtTOHMM 

■jacEjcM!' 

■•>i;!:l:Eg 

■ijclUiMd 

■!'>!K!l.mi 

IIHH 

bhh 

IHIH 

H^m 

im^i 

ig!wiinin«t«Ri 

wtiiiin 

mean 

KiMiKiy 

Location 

■•y/AiiEh 

IHHHI 

IHHH 

hihi 

■■■ 

. . Ill  inm— 

25  Data  Points 

litPiagmidtiHBiimasM 

immaidtiMimaM 

Set1 

Set2 

Rep1 

isnsHi 

■l.-l.-MKHa 

Shape 

■D^S 

■tUbk 

H!l!l!l!£ 

Ki!i!;!n 

■l!li!:i!li!:H 

Scale 

■fcWlAEI 

mi!V:Piu 

BjSEHI 

2500 

m:>VrXA>>iA 

Location 

■riEMEill 

0 

iryWurtM 

■KtgE! 

■feW:W«I:!il 

KbU-yJkWtl 

KiKMIiK 

mm 

mii^B 

UiAkliirkU 

«»!:!!«! 

P>ttKK*i 

m-ihlWiltK 

PXllPVi!t\ 

PZtkK/rK 

ia:r/jtM.ia 

1 

I. 

2985.502 

3757.219 

3917.9257 

■c):Wl:tiei 

mmxwM 

■ciUIAtH 

gjgfJWAH 

mWfJrkVIA 

gyjiEfcTiEM 

g«!.TBKHKl 

mAVitViU 

»5t=MHn:| 

eiitKEEfc] 

HHHi 

bhh 

mm 

mHB 

HHHH 

IBHH 

■■icg5K.-Mj 

K««MH 

mm 

■mm 

■ap*«=M 

ISK-KSH 

■*KK=I.U 

HHH 

HHH 

HHH 

HIHH 

IHHI 

HHHI 

mBH 

»iK*E!=H 

WMK^icU! 

K^MEEII 

HHi 

HBHi 

HHH 

HHHI 

■■■■ 

HIHHI 

■IBB 

IDFR  Failure 

I^H^H 

IHHH^^^^I 

i^B^H 

i^Bmn 

IHHHI 

IHH^^H 

'Hi^m 

IHHH 

ni^i^m 

TRUE  DFRPARAMI 

ll^lflI7:1Jild 

(Top  Weibull-i-+  Selection) 

(Weibull-r-i-  Exponential) 

Weibull 

|5  Data  Points 

High  Level  Fitting  Parameters 

Low  Level  Fitting  Parameters 

0.95 

Setl 

Set2 

Repi 

Rep2 

Rep3 

Repi 

Rep2 

Rep3 

Scale 

868.5596 

698.6931 

169.6434 

Shape 

2.3023 

0.6745 

0.8935 

Lambda 

0,0007 

0.0003 

0.0004 

Location 

0 

1516.3796 

1366.5096 

401.407 

Scale 

2553.192 

2135.705 

2627.934 

mean 

1428.571 

3333.333 

H^^3 

2218.6249 

1700.3733 

2722.0258 

Location 

0 

658.6182 

0 

Location 

652.0483 

0 

2697.295 

2792.1665 

3234.4815 

3966.098 

10702.2462 

7299.204 

Failure  PDF 

(T  op  Weibull+-»-  Selection) 

(Weibulk-f  Exponential) 

25  Data  Points 

High  Level  Fitting  Parameters 

Low  Level  Fitting  Parameters 

Setl 

Set2 

Set3 

Fiepi 

Rep2 

FiepS 

Repi 

Rep2 

Rep3 

207.8036 

42.6759 

26.6541 

Shape 

0.9439 

0.7877 

0.8449 

Lambda 

0.0003 

0.0003 

0.0003 

266.2007 

110.6362 

73.5251 

Scale 

3714,093 

2702.41 

3310.37 

mean 

3333.333 

3333.333 

3333.333 

564.8699 

125.2635 

202.0481 

Location 

126.86 

21.438 

0 

Location 

0 

0 

0 

571.728 

367.1112 

481.4955 

575.5735 

412.2275 

578.7774 

956.9291 

417.0253 

911.8075 

1372.7997 

514.4869 

958.3334 

1458.9632 

646.2646 

1024.3937 

1807.3052 

905.7708 

1215.1514 

2216.5446 

937.2157 

1340.3025 

2250.8431 

1167.433 

1446.7933 

2279.4159 

1336.8557 

1703.5091 

2631 .9957 

1660.6554 

1726.7708 

3200.9254 

1813.3288 

2022.7823 

3607.7149 

2070.6515 

2047.3684 

3808.4165 

2174.3106 

2517.398 

4135.8181 

2720.8299 

2566.4763 

4219.2527 

3012.5395 

5004,063 

4349.9903 

4941.1598 

5040.78 

4892.6842 

5374.681 6 

6443.4536 

5591 .2831 

6328.194 

6489.4017 

6274.9309 

8346.4754 

10469.55 

9276.5345 

8829.5878 

10626.242 

15164.2679 

10020.3126 

11879.798 

16880.6085 

13523.2832 

13441.007 
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Component  4: 


Failure  PDF 

KTop  Weibull+-f  Selection)  I 

(Weibu}I++  Exponential) 

5  Data  Points 

High  Level 

Fitting  Parameters 

Low  Level  FKtinq  Parameters 

Set1  Set2 

1109.878  814.5934 

Sets 

Shape 

,Rep1 

1.3709 

Rep2 

6.5579 

Rep3 

Lambda 

Rep1 

0.001 2 

Rep2 

0.0007 

Rep3 

0.0024 

1321.429  1365.19 

Scale 

995.1 192 

1476.695 

mean 

833.3333 

1428.571 

416.6667 

1895.723  1475.554 

Location 

971.24 

0 

Location 

1040.31 

0 

458.4051 

2216.109  1517.455 

2873.409  1664.4 

Ex 

p.  Lambda 

0.0024 

mean 

416.6667 

Location 

458.4051 

TRUE  IFRPARAM 
Webull  I 


Failure  PDF 
25  Data  Points 


(Top  Weibull++  Selection) 
High  Level  Rtting  Parameters 


(Weibull++  Exponential) _ 

Low  Level  Fitting  Parameters 


Setl 

Set2 

Set3 

Repl 

Rep2 

Rep3 

Repl 

Rep2 

Rep3 

530.2771 

200.3715 

575.2981 

Shape 

1.298 

1.5358 

1.5218 

Lambda 

0.0008 

0.0007 

0.0009 

540.401 

726.1561 

248.5755 

359.6599 

835.6282 

848.035 

Scale 

Location 

1501.29 

440.23 

1741.393 

0 

1347.52 

483.3504 

mean 

Location 

1250 

530.2771 

1428.571 

200.3715 

1111.111 

575.2981 

760.3249 

426.0651 

866.1 735 

840.5436 

434.9741 

915.2016 

944.0206 

554.9306 

1048.103 

979.1587 

579.127 

1054.785 

1069.765 

688.181 

1079.655 

1075.722 

959.4233 

1129.904 

1153.189 

996.8733 

1156.848 

1153.868 

1140.553 

1350.402 

1375.649 

1432.618 

1367.315 

1501.503 

1493.902 

1469.085 

1627.924 

1678.524 

1596.448 

1919.104 

1869.419 

1624.494 

2285.03 

1912.531 

1842.271 

2482.025 

2045.89 

1905.994 

2566.556 

2082.82 

2044.315 

2708.679 

2267.416 

2119.071 

2774.881 

2288.218 

2497.592 

2938.419 

2367.334 
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Components  14, 15, 16  (Identical): 
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Components  18, 19, 20  (Identical): 
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Weibulk+  hXDonential 
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Appendix  E:  Data  Fitting  Graphs 


Examples  of  True  versus  Weibull++  Wizard  and 
Exponential  Fitted  Distributions 
for  Component  1  (Final  Experiment): 

IFR  Failure  PDF  (Weibull) 

5  data  points 
Replication  3 


IFR  Failure  PDF  (Weibull) 
25  data  points 
Replication  3 
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DFR  Failure  PDF  (Weibull) 
5  data  points 
Replication  3 


DFR  Failure  PDF  (Weibull) 
25  data  points 
Replication  3 
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Repair  PDF  (Lognormal) 
5  data  points 
Replication  3 


Repair  PDF  (Lognormal) 
25  data  points 
Replication  3 


Appendix  F: 


Birnbaum  Structural  Component  Importance 
Measure  Results  for  Final  Experiment 


Small  /  Series-Parallel  Structure:  Small  /  Complex  Structure: 
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Large  /  Series-Parallel  Structure:  Large  /  Complex  Structure: 
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Appendix  G:  Multivariate  Analysis  of  RAPTOR  Output 


1.  ANALYSIS  TECHNIQUES 

Overview 

A  main  objective  of  this  study  was  to  provide  insight  for  the  reliability  community  in 
assessing  differences  in  various  systems  of  components  through  multivariate  analysis  of 
simulation  output.  Several  multivariate  techniques  were  applied,  including  discriminant 
analysis  (DA),  neural  networks,  logistic  regression,  principal  component  analysis  (PCA), 
and  factor  analysis  (FA). 

Discriminant  Analysis  (DA) 

A  primary  analysis  objective  was  to  discriminate  between  large  versus  small,  complex 
versus  series-parallel,  and  increasing  failure  rate  (IFR)  versus  decreasing  failure  rate 
(DFR)  component  structures.  Discriminant  analysis  was  the  key  method  to  achieve  this 
objective.  Due  to  the  relatively  small  size  of  the  data  set,  the  discriminant  function  was 
formed  from  the  entire  data  set.  Therefore,  true  validation  cannot  occur  until  the 
discriminant  function  is  tested  against  future  observations.  As  will  be  discussed  later,  the 
formatting  of  the  data  was  a  major  difficulty  in  conducting  discriminant  analysis.  Because 
of  this,  and  as  a  learning  exercise,  DA  was  attempted  on  different  forms  of  the  data  set, 
namely  standardized  data  and  transformed  data  (using  a  Box-Cox  transformation). 
Furthermore,  since  the  variance-covariance  matrices  were  only  statistically  equal  for  the 
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IFR  versus  DFR  case,  discriminant  functions  were  calculated  using  the  within-class 
covariance  matrices  in  addition  to  using  the  pooled  matrices  (for  the  large  versus  small  and 
complex  versus  series-parallel  cases).  This  was  done  mostly  as  a  learning  exercise  to  see 
what  would  happen  and  if  any  differences  would  occur  in  the  discriminant  results.  In 
general,  as  detailed  in  the  results  section  of  this  paper,  significant  success  was  achieved  in 
discriminating  between  classes  in  all  3  cases. 

Neural  Networks 

Since  a  quadratic  discriminant  function  was  the  most  effective  for  the  complex  versus 
series-parallel  case,  a  neural  network  was  also  employed  to  assess  it’s  ability  to 
discriminate  between  complex  and  series-parallel  component  structures.  The  neural  net 
was  trained  on  standardized  data  using  back-propagation  and  sigmoidal  processing  with 
one  hidden  layer  containing  20  nodes.  A  ‘full’  neural  net  was  run  using  all  the  variables  as 
well  as  a  ‘reduced’  net  containing  only  the  3  most  salient  variables.  Good  discriminant 
success  was  achieved  (consistent  with  the  DA  results)  for  the  training  and  validation  sets 
for  both  the  full  and  reduced  models. 

Logistic  Regression 

As  an  additional  exercise,  logistic  regression  was  also  tried  in  an  attempt  to 
discriminate  between  complex  and  series-parallel  component  stmctures.  The  models 
included  a  full  model  logistic  regression  of  raw,  standardized,  and  transformed  data, 
without  success.  The  software  used  in  the  logistic  regression  analysis  (SAS  and 
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IMP)  could  only  produce  a  viable  regression  model  on  a  reduced  set  of  variables  (the  3 
most  salient  variables  identified  in  the  neural  net  analysis  were  used).  Logistic  regression 
proved  to  be  the  least  powerful  method  for  discriminating  between  complex  and  series- 
parallel  component  structures. 

Principal  Component  Analysis  (PCA) 

Another  analysis  objective  was  to  see  if  the  majority  of  output  variance  could  be 
adequately  explained  in  smaller  dimensions.  To  achieve  this  objective,  principal 
component  analysis  (PCA)  based  on  the  correlation  matrix  was  conducted.  Al±ough  the 
loading  structure  was  not  completely  clear-cut,  by  using  Kaiser’s  criterion  a  reduction  in 
the  dimensionality  of  the  data  set  to  3  components  was  achieved  which  explained  a 
majority  (82%)  of  the  output  data  variance.  Some  success  in  discriminating  between  large 
versus  small  and  IFR  versus  DFR  structures  using  component  score  rankings  was  also 
achieved. 

Factor  Analysis  (FA) 

Our  final  analysis  objective  was  to  identify  possible  common  underlying  factors  with 
common  variance.  Using  factor  analysis  with  varimax  data  rotation,  3  underlying  factors 
were  identified.  The  rotation  produced  much  more  clearly  defined  factor  loadings.  As 
with  PCA,  some  success  was  achieved  in  discriminating  between  large  versus  small  and 
BFR  versus  DFR  stmctures  using  factor  score  rankings. 
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II.  DATABASE 


General  Description 

Multivariate  analysis  was  conducted  on  output  data  produced  by  system  component 
reliability  models  developed  and  run  on  the  Rapid  Availability  Prototyping  for  Testing 
Operational  Readiness  (RAPTOR)  software.  RAPTOR,  created  by  HQ  AFOTEC/SAL, 
creates  reliability,  maintainability,  availability  (RM&A)  and  sparing  models  for  various 
systems  undergoing  operational  test  and  evaluation  (OT&E). 

Specific  Output  Measures 

The  specific  output  measures  analyzed  are  defined  below: 

Availability:  The  ratio  of  the  time  the  system  is  up  (operational)  versus  total 
simulation  time. 

Mean  Time  Between  Downing  Events  (MTBDE):  The  average  time  between 
events  which  bring  the  entire  system  down. 

Mean  Down  Time  (MDT):  The  average  amount  of  time  the  entire  system  is 

down. 

Mean  Time  Between  Maintenance  (MTBM):  The  average  amount  of  time 
between  any  maintenance  actions  performed  on  any  components  of  the  system. 

Mean  Repair  Time  (MRT):  The  average  amount  of  time  it  takes  to  repair  any 
component  in  the  system. 

Analysis  on  the  standard  deviations  of  all  of  the  above  simulation  output  measures  was 
also  conducted. 
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Thirty-eight  different  system  models  with  various  characteristics  were  created  and  run  on 
RAPTOR  for  a  duration  of  50,000  simulation  time  units  per  run.  The  three  characteristics 
which  define  each  system  of  components  are  structure  type,  failure  probability  density 
function  (pdf)  type,  and  system  size.  The  breakdown  for  each  category  is  as  follows: 

-  Structure  Type:  Complex  (non  series-parallel)  or  Series-Parallel  network 

-  Overall  Component  Failure  pdf  Type:  Increasing  Failure  Rate  (IFR)  or 
Decreasing  Failure  Rate  (DFR) 

-  Size:  Large  (20  components)  or  Small  (5  components) 

Two  examples  of  structure  types  used  in  the  study  are  shown  in  Figures  F-1  and  F-2. 


Figure  F-1.  Small  /  Complex  Structure  Type 
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Figure  F-2.  Large  /  Series-Parallel  Structure  Type 
Twelve  basic  structures/systems  were  developed:  3  large  complex  systems,  3  large  series- 
parallel  systems,  3  small  complex  systems,  and  3  small  series-parallel  systems.  The 
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parameters  of  the  Weibull  distributed  failure  rates  for  specific  components  in  each  system 
were  varied,  and  10  runs  for  each  configuration  measuring  the  outputs  described  above 
(averaged  over  the  10  simulation  runs)  were  conducted.  When  re-configuring  a 
component  failure  rate  from  IFR  to  DFR,  the  same  average  failure  rate  was  maintained  by 
adjusting  the  Weibull  scale  parameter.  Therefore,  when  a  component  is  altered  from  IFR 
to  DFR  (or  vice  versa),  the  only  thing  that  changes  is  the  fact  that  it’s  failure  distribution  is 
changed  from  Weibull  IFR  to  Weibull  DFR.  Some  runs  were  conducted  with  all 
component  failure  pdf’s  either  exclusively  IFR  or  DFR,  and  some  were  run  where  20%  of 
the  component  failure  distributions  were  altered  to  the  opposite  type.  To  simplify  the 
analysis,  any  system  which  had  a  predominant  (80%  or  more)  component  failure 
distributions  of  IFR  or  DFR,  was  labeled  as  IFR  or  DFR,  respectively.  The  final  result 
was  38  total  configurations.  An  entire  overview  of  the  structure  types  and  simulation 
outputs  is  provided  in  Table  F-1. 
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Table  F-1.  RAPTOR  Output  Database 
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III.  ANALYSIS  OBJECTIVES 


Purpose  of  Investigation 

The  purpose  of  the  investigation  was  to: 

1)  Ascertain  whether  one  can  distinguish  between  the  complex  versus  series- 
parallel  structures,  IFR  versus  DFR  configurations,  and  large  versus  small  system  sizes 
based  on  the  simulation  outputs; 

2)  Identify  which  output  measures  provide  the  most  discriminant  power; 

3)  See  if  one  can  adequately  explain  a  majority  of  the  output  variance  in  smaller 
dimensions;  and 

4)  Identify  possible  common  underlying  factors  with  common  variance. 

Variables  Used 

All  10  RAPTOR  output  variables  were  used  in  the  analysis.  In  some  cases,  nearly 
equivalent  results  could  be  obtained  by  only  using  the  most  salient  variables  (this  will  be 
discussed  in  more  detail  in  the  results  section  of  this  report).  Since  there  is  a  large 
disparity  in  magnitudes  of  the  output  variables,  the  variance-covariance  matrix  was  sparse 
(contained  many  zeros).  To  alleviate  computational  problems  resulting  from  this, 
standardized  data  was  used  for  most  analyses.  The  standardized  data  set  is  depicted  in 
Table  F-2. 

When  checking  for  multivariate  normality  for  discriminant  analysis,  several  of  the 
variables  did  not  pass  the  Shapiro- Wilk  test  for  normality  (at  a  10%  level  of  significance). 
In  an  attempt  to  achieve  multivariate  normality,  a  Box-Cox  transformation  was  conducted 
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on  all  variables.  The  affects  of  the  Box-Cox  transformation  on  the  passage  of  the  Shapiro- 


Wilk  test  for  each  variable  are  shown  in  Table  F-3. 
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Table  F-2.  Standardized  Data  Set 


1  i  Lar 

ge 

Small 

Complex 

S-P 

IFR  I 

DFR  1 

Variable  ! 

Before 

After 

Before  j 

After 

Before 

After 

Before 

After 

Before 

After  I 

Before 

After 

Ao  j 

Fail 

Fail 

Fail  i 

Fail 

Pass 

Pass 

Pass 

Pass 

Fail 

Faii  1 

Pass 

Pass 

Ao  S.D.  1 

Pass 

Pass 

Pass  I 

Pass 

Pass 

Pass 

Pass 

Pass 

Pass 

Pass  I 

Pass 

Pass 

MTBDE  i 

Fail 

Pass 

Fail 

Pass 

Fail 

Pass 

Fail 

Pass 

Fail 

Pass  ; 

Fail 

Pass 

MTBDE  S.D.  j 

Fail 

Pass 

Fail 

Pass 

Fail 

Pass 

Fail 

Pass 

Fail 

Pass  i 

Fail 

Pass 

MDT  i 

Pass 

Pass 

Fail  1 

Fail 

Fail 

Pass 

Fail 

Fail 

Pass 

Pass  i 

Pass 

Pass 

MDTS.D.  I 

Pass 

Pass 

Pass  1 

Pass 

Fail 

Pass 

Pass 

Pass 

Pass 

Pass  i 

Pass 

Pass 

MTBM  i 

Fail 

Pass 

Pass  i 

Pass 

Fail 

Fail 

Fail 

Fail 

Fail 

Fail  i 

Fail 

Fail 

MTBMS.D.  i 

Pass 

Pass 

Fall 

Pass 

Fail 

Pass 

Fail 

Pass 

Fail 

Fail  i 

Fail 

Fail 

MRT  i 

Fail 

Fail 

Pass  i 

Pass 

Pass 

Pass 

Fail 

Fail 

Pass 

Pass  I 

Pass 

Pass 

MRT  S.D.  i 

Pass 

Pass 

Pass  I 

Pass 

Pass 

Pass 

Faii 

Bnsssi 

Faii 

Pass  j 

Pass 

Pass 

*  Boldface  cells  note  where  improvement  was  achieved 


Table  F-3.  Effects  of  Box-Cox  Transformation  on  Shapiro-Wilk  Normality  Test  for  Each  Variable 
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From  Table  F-3,  it  is  apparent  that  an  improvement  in  the  overall  normality  of  the  data 
was  achieved.  Although  not  all  variables  passed  the  Shapiro-Wilk  test  after  the 
transformation,  the  majority  of  the  variables  did  pass.  Therefore,  the  assumption  of 
multivariate  normality  was  reasonably  justified  for  use  in  discriminant  analysis. 
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IV.  ANALYSIS  RESULTS 


Special  Problems  Encountered 

The  most  difficult  problem  encountered  was  the  formatting  of  the  data.  As  discussed 
previously,  the  large  scale  differences  in  the  data  caused  numerical  problems,  but  this  was 
overcome  via  standardization.  Another  problem  was  the  lack  of  multivariate  normality, 
which  was  addressed  by  the  use  of  Box-Cox  transformations.  In  the  end,  several  different 
data  formats  were  tried  (raw,  standardized,  and  transformed)  in  the  discriminant  analysis 
to  see  what  type  of  results  would  be  achieved  with  each  format. 

When  conducting  logistic  regression,  SAS  and  IMP  could  not  produce  a  viable 
regression  model  using  all  variables.  This  was  true  using  the  raw  simulation  output  data, 
standardized  data,  as  well  as  transformed  data.  However,  a  viable  model  was  produced 
when  the  set  of  variables  was  reduced  (down  to  3)  to  those  that  were  identified  as  most 
salient  in  the  neural  network  analysis. 

Another  problem  was  the  difficulty  in  interpreting  the  principal  components.  A  ‘clean’ 
separation  in  the  principal  component  loadings  was  not  apparent,  making  the  analysis 
challenging.  Although  principal  components  were  defined  from  this  analysis,  the 
interpretation  may  be  subject  to  debate  due  to  the  ambiguity  in  component  loadings. 
However,  after  varimax  rotation  of  the  data,  a  much  clearer  loading  structure  was  revealed 
in  the  subsequent  factor  analysis. 

Discrimination  Between  Categories  of  Component  Structures 

Several  multivariate  techniques  were  used  in  an  attempt  to  discriminate  between  large 
versus  small,  complex  versus  series-parallel,  and  increasing  failure  rate  (DPR)  versus 
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decreasing  failure  rate  (DFR)  component  stmctures;  DA,  neural  nets,  logistic  regression. 


as  well  as  score  rankings  resulting  from  PCA  and  FA.  The  overall  discriminant  results  for 


all  methods  are  shown  in  Table  F-4  for  direct  comparison. 


Discriminant  Results 

(percentages  show  classification  accuracy) 

Data 

Method 

Complex/S-P 

IFR/DFR 

Standardized 

SAS  Pooled 

100%  / 100% 

94%/ 85% 

95%/ 95% 

SAS  Pooled 

100%  / 100% 

94%/ 90% 

89%/ 95% 

Transformed 

SAS  Unpooled 

100%  / 100% 

100  %  / 100% 

____ 

JMP 

100%  / 100% 

94%/ 90% 

89%/ 95% 

Full  Neural  Net:  Training 

— 

93%  /  100  % 

____ 

Full  Neural  Net:  Validation 

— 

100%  / 100% 

Standardized 

Reduced  Neural  Net:  Training 

— 

98%  /  100  % 

Reduced  Neural  Net:  Validation 

100%  / 100% 

Reduced  Logistic  Regression 

— 

67%/ 85% 

____ 

Raw 

Component  Score  Ranking 

89%/ 90% 

— — 

84%/ 74% 

Factor  Score  Ranking 

100%  / 100% 

84%/ 95% 

Best  Discriminant  Function 

Linear 

Quadratic 

Linear 

MTBDE 

MRTSD 

MTBM 

Ao 

AoSD 

Best  Discriminant  Variable(s) 

MRT 

MTBM  SD 

MDT 

MDTSD 

Table  F-4.  Classification  Accuracy  for  all  Methods  Used  for  Discrimination 
For  the  most  part,  the  results  were  consistent  across  methods  with  excellent 
discriminant  success.  There  was  strong  agreement  between  methods  on  which  variables 
served  as  the  best  discriminants  (e.g.  discriminant  loadings,  neural  net  salient  variables, 
and  components/factors  which  best  discriminated  for  each  category  showed  strong 
agreement).  This  general  consistency  across  methods  provided  greater  confidence  in  the 
overall  analysis.  The  classification  accuracy  percentages  for  DA  may  be  inflated  because 
the  entire  data  set  was  used.  Logistic  regression  proved  to  be  the  weakest  discriminant 
tool  in  the  complex  versus  series-parallel  case. 
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Neural  Net  Results 

To  help  identify  the  variables  which  contributed  most  in  discriminating  between  classes 
in  the  neural  net,  several  graphical  outputs  produced  by  the  Statistical  Neural  Network 
Analysis  Package  (SNNAP)  software  were  reviewed.  As  an  example,  the  following 
derivative  graphs  help  show  which  variables  had  the  greatest  discriminant  power.  Looking 
at  Figure  F-3,  the  graphs  with  the  more  ‘pointed’  curves  identify  the  more  salient  variables 
(Ao,  MDT,  and  MRT). 


Figure  F-3.  Neural  Net  Derivative  Saliency  Graphs 
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Reduction  in  Dimensionality  (PCA) 


The  objectives  of  performing  a  PCA  on  the  database  were  to  reduce  the  dimensionality 


of  the  data  and  to  further  attempt  to  discriminate  between  stmcture  (by  type,  failure  pdf, 


and  size).  Due  to  the  difference  in  the  units  of  the  data,  the  PCA  was  performed  using  the 


data’s  correlation  matrix  (see  Table  F-5). 


Variable 

Ao 

Ao 

S.D. 

MTBDE 

MTBDE 

S.D. 

MDT 

MDT 

S.D. 

MTBM 

MTBM 

S.D. 

MRT 

MRT 

S.D. 

Ao 

1 

-0.318 

0.7206 

0.6327 

-0.5495 

0.0577 

-0.332 

-0.2459 

-0.658 

-0.3873 

AoS.D. 

-0.3177 

1 

-0.371 

-0.2081 

0.1285 

0.4976 

0.2352 

0.5285 

0.344 

0.692 

MTBDE 

0.7206 

-0.371 

1 

0.9219 

-0.1915 

0.1148 

-0.173 

-0.1442 

-0.472 

-0.2649 

MTBDE  S.D. 

0.6327 

-0.208 

0.9219 

1 

-0.3184 

0.169 

-0.255 

-0.1122 

-0.41 

-0.1986 

MDT 

0.1285 

-0.192 

-0.3184 

1 

0.0143 

0.8225 

0.5198 

0.544 

0.4617 

MDT  S.D. 

0.0577 

0.4976 

0.1148 

0.169 

0.0143 

1 

0.1952 

0.455 

0.268 

0.5232 

MTBM 

-0.3315 

0.2352 

-0.173 

-0.2554 

0.8225 

0.1952 

1 

0.7034 

0.547 

0.5575 

MTBMS.D. 

-0.2459 

0.5285 

-0.144 

-0.1122 

0.5198 

0.455 

0.7034 

1 

0.549 

0.7272 

MRT 

-0.6576 

0.3435 

-0.472 

-0.4099 

0.5439 

0.2676 

0.5474 

0.5487 

1 

0.5559 

MRTS.D. 

-0.3873 

0.692 

-0.265 

-0.1986 

0.4617 

0.5232 

0.5575 

0.7272 

0.556 

1 

Table  F-5.  Data  Correlation  Matrix 


IMP  software  calculated  the  principle  components.  Three  components  were  retained 


based  on  Kaiser’s  criterion.  As  Table  F-6  indicates,  these  components  accounted  for 


81.85%  of  the  data  set  variation. 


4.6365 

2.19 

1363 

0.5431 

0.4333 

0.3292 

0.2094 

0.1776 

0.101 

0.0219 

Percent: 

46.3649 

21.859 

13.626 

4.3333 

3.2919 

2.0938 

1.7758 

1.006 

0.2187 

Cum  Percent 

46.3649 

68.224 

81.85 

87.2808 

91.614 

94.906 

96.9998 

98.7756 

99.78 

100 

Table  F-6.  Component  Eigenvalues  and  Percentages 


Using  the  eigenvalues  and  eigenvectors  (eigenvector  multiplied  by  the  square  root  of  the 


corresponding  eigenvalue),  JMP  calculated  the  loadings  matrix.  As  shown  in  Table  F-7, 


only  the  first  three  loadings  were  analyzed. 
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AvaflabUity 

■0.7264 

0.49766 

0.01967 

Ao  S.D. 

-0.59 

MTBDE 

-0.607411 

0.6781 

0.35319 

MTBDES.D. 

-0.572701 

0.7072 

0.16203 

MDT 

0.71309 

-0.01032 

0.6206 

MDT  S.D. 

0.33791 

0.6865 

-0.3972 

MTBM 

0.74412 

0.21079 

0.52521 

MTBM  S.D. 

0.75028 

0.47291 

0.0949 

MRT 

0.81152 

-0.05729 

0.067 

MRT  S.D. 

0.79726 

0.39407 

-0.1759 

Table  F-7.  PCA  Loadings  Matrix 

After  careful  examination  of  the  above  loading  matrix,  in  conjunction  with  knowledge  of 
the  database,  each  component  was  labeled  based  on  the  bold  numbers  in  the  respective 
column  of  the  matrix. 

•  Component  1  Maintenance  Index 

•  Component  2  Deviation  Down  Time  Index 

•  Component  3  Down  Time  Average  Index 

After  successfully  reducing  the  dimensionality  of  the  database  from  ten  to  three, 
component  scores  were  calculated  to  see  if  they  were  effective  at  discriminating  a  given 
structure  into  the  following  attributes: 

•  Type:  Complex  or  Series-Parallel 

•  Failure  pdf:  Increase  Failure  Rate  (IFR)  or  Decreasing  Failure  Rate  (DFR) 

•  Size:  Large  or  Small 

Each  vector  of  component  scores  was  sorted  in  descending  order  to  look  for  a  pattern. 
The  noticeable  patterns  appear  in  Table  F-8. 
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Component  2 

Size 

Score 

Score 

Small 

3.9875798 

DFR 

4.2453607 

Small 

3.27563 

IFR 

2.7344832 

Small 

3.1916125 

DFR 

2.6642524 

Small 

2.5766016 

DFR 

Small 

2.5624751 

IFR 

1.9337528 

Small 

2.3625727 

DFR 

1.4825866 

Small 

1.8048802 

DFR 

1.4374841 

Small 

1.7765076 

DFR 

1.0900088 

Small 

1.5808509 

DFR 

0.9908706 

Small 

1.5494862 

DFR 

0.6643567 

Small 

1.5341994 

DFR 

0.622185 

Small 

1.256253 

IFR 

0.5673482 

Small 

1.2471147 

DFR 

0.354911 

Small 

1.0250701 

DFR 

0.2859006 

Small 

0.6167541 

DFR 

0.2601498 

0.5833746 

DFR 

0.2469027 

II  Small 

0.4965454 

DFR 

0.0693093 

0.4641742 

IFR 

-0.134626 

Small 

0.4333046 

IFR 

-0.141776 

Small 

0.2313683 

DFR 

-0.239745 

0.2225142 

IFR 

-0.251629 

1  Small 

-0.096977 

IFR 

-0.376745 

■1S9 

-0.292476 

DFR 

-0.504691 

IFR 

-0.505771 

1  Small  1 

-0.451488 

IFR 

IFR 

-1.330643 

IFR 

-0.847647  II 

-1.357726 

IFR 

Mit:l‘l‘ltlcfeWll 

-1.53975 

IFR 

-1.165003 

-1.637563 

DFR 

-1.242413 

-1.977937 

DFR 

-1.399575 

-2.244068 

IFR 

-1.485876 

-2.327493 

IFR 

-1.587855 

-2.39671 

IFR 

-1.683993 

-3.35344 

IFR 

-1.697677 

-3.691377 

DFR 

-1.879922 

-4.077877 

IFR 

-2.083892 

-5.058193 

IFR 

-2.153662 

Table  F-8.  Component  Scores 


Even  though  the  component  scores  do  not  discriminate  completely,  there  appears  to  be 
some  usefulness  in  these  scores  in  determining  the  attributes  of  a  given  structure  using  the 
following  formulas: 


•  If  Component  1  Score  ^  0  ^ 

•  If  Component  1  Score  <  0 

•  If  Component  2  Score  ^  0 

•  If  Component  2  Score  <  0  ^ 


Classify  the  Structure  as  Small 
Classify  the  Structure  as  Large 
Classify  the  Structure  as  DPR 
Classify  the  Structure  as  IFR 


The  component  score  3  did  not  appear  to  have  any  discriminating  power. 
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Identification  of  Underlying  Factors  (FA) 


Factor  analysis  was  performed  on  the  database  for  two  reasons:  to  identify  any 


possible  underlying  factors  and  to  use  these  factors  to  discriminate  between  the  attributes 
of  a  given  structure.  Using  SAS  and  varimax  rotation,  a  rotated  factor  pattern  was 
obtained.  As  can  be  seen  in  Table  F-9,  the  underlying  factors  fell  out  very  well. 


Factor  1 

Factor  2 

Factor  3 

Avail 

0.79513 

-0.37048 

-0.07841 

Ao  S.D. 

-0.30561 

0.02208 

0.84431 

MTBDE 

0.9701 

-0.01508 

-0.11017 

MTBDE  S.D. 

0.91457 

-0.12722 

0.04128 

MDT 

-0.21567 

0.92026 

-0.01965 

MDT  S.D. 

0.21902 

0.05537 

0.832 

MTBM 

-0.09176 

0.91097 

0.18895 

MTBM  S.D. 

-0.02499 

0.64731 

0.61313 

MRT 

-0.47654 

0.57019 

0.33782 

MRTS.D. 

0.46513 

0.75333 

Table  F-9.  Rotated  Factor  Pattern  (from  SAS  with  Varimax  Rotation) 

•  Factor  1  Functionality 

•  Factor  2  Repair 

•  Factor  3  Variance 

The  common  variance  contributions  for  each  factor  can  be  seen  in  Figure  F-4. 


Functionality 


Repair 


Figure  F-4.  Common  Variance  Contributions  by  Factor 
Using  standardized  data,  the  factor  scores  were  obtained  for  each  of  the  three  factors. 
As  with  PCA,  an  attempt  was  made  to  discriminate  a  given  structure  by  one  of  its  three 
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attributes  by  sorting  each  factor  score  in  descending  order.  As  seen  in  Table  F-10,  factor 


scores  2  and  3  were  very  good  at  discriminating  respectively  between  structure  size  and  its 


failure  rate  pdf. 


Size 

Factor  2 

Failure  pdf  Type 

Factor  3 

Small 

1.754919849 

DFR 

2.375509734 

Small 

1,262692335 

DFR 

2.108713858 

Small 

1.257212778 

IFR 

1.879361215 

Small 

1.248601501 

DFR 

1.412724627 

Small 

1.198436102 

DFR 

1.338927349 

Small 

1.171639447 

DFR 

1.319662935 

Small 

1.160513013 

DFR 

0.809561313 

Small 

1.148780849 

DFR 

0.752946945 

Small 

1,070286433 

DFR 

0.476564177 

Small 

1.041964654 

IFR 

0.402846391 

Small 

0.803497969 

DFR 

0.39528509 

Small 

0.781802256 

DFR 

0.351588765 

Small 

0.708731145 

DFR 

0.312586852 

Small 

0.645365044 

IFR 

0.298188266 

Small 

0.636668185 

DFR 

0.287840003 

Small 

0.616611494 

DFR 

0.235440949 

Small 

0.581937118 

DFR 

0.217068964 

Small 

0.058035318 

DFR 

0.0410742 

Small 

0.045624986 

DFR 

Small 

mKmtmm 

DFR 

MFitlrifHEa 

.Large 

mmmmm 

DFR 

■MHIZEiriMi 

IFR 

mnAnmiiim 

IFR 

-0.357517609  I 

HISSI 

1 

1 

\ 

\ 

EFR 

-0.796558728 

DFR 

-0.52311732  1 

IFR 

EFR 

-0.652270592 

-0.917883041 

EFR 

-0.658484054 

IFR 

-0.728537638 

-0.975227776 

IFR 

-0.751696689 

-1.066756545 

IFR 

-0.788207166 

mnmsmm 

IFR 

-0.84117866 

HE53I 

-1.104270837 

IFR 

-0.871777043 

H1S3I 

-1.109476158 

IFR 

-1.021398497 

-1.301022087 

EFR 

-1.239891531 

-1.306582726 

IFR 

-1.303301235 

Large 

-1.327933971 

IFR 

-1.459488901 

Large 

-1.523803607 

IFR 

-2.00028026 

Table  F-10.  Factor  Scores 


•  If  Factor  Score  2  >  -0.15 

•  If  Factor  Score  2  <  -0. 15 

•  If  Factor  Score  3  ^  -0.30 

•  If  Factor  Score  3  <  -0.30  ^ 


Classify  the  Structure  as  Small 
Classify  the  Structure  as  Large 
Classify  the  Structure  as  DFR 
Classify  the  Structure  as  IFR 


Factor  score  1  did  not  appear  to  have  any  discriminating  power. 
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Insights 

Several  useful  conclusions  can  be  drawn  from  this  study.  First,  it  was  demonstrated 
(using  a  moderately  small  sample  size)  that  successful  discrimination  can  occur  between 
large  versus  small,  complex  versus  series-parallel,  and  IFR  versus  DFR  component 
stmctures  based  on  RAPTOR  simulation  output.  All  multivariate  techniques 
demonstrated  were  moderately-to-highly  successful  in  discriminating  between  the  defined 
classes.  Through  the  discrimination  analysis,  it  was  discovered  that  predominantly  DFR 
stmctures  display  a  relatively  higher  simulation  output  variability.  Therefore,  RAPTOR 
availability  model  output  variability  serves  as  a  good  discriminant  for  EFR  versus  DFR 
stmctures.  Furthermore,  Mean  Time  Between  Maintenance  (MTBM)  is  an  excellent 
discriminant  variable  for  the  large  versus  small  stmcture  classification  case.  This 
conclusion  makes  intuitive  sense,  since  one  would  expect  a  decrease  in  the  average  time 
between  maintenance  actions  on  components  as  the  number  of  components  in  the 
stmcture  increases.  The  analysis  provides  empirical  support  to  this  intuitive  assessment. 
Additionally,  it  was  discovered  that  neural  nets  can  be  used  to  effectively  discriminate 
when  the  discriminant  function  may  be  of  a  higher  order. 

Additionally,  the  analysis  revealed  that  the  RAPTOR  simulation  output  variance  can 
be  explained  via  3  principal  components:  a  maintenance  index,  a  deviation  down  time 
index,  and  a  down  time  average  index.  A  majority  of  the  output  variance  (82%)  is 
explained  by  these  three  components.  By  using  a  rank  order  of  the  maintenance  index 
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(component  1)  scores  and  deviation  down  time  index  (component  2)  scores,  reasonable 
discrimination  between  large  and  small  structures,  and  IFR  and  DFR  structures 
respectively,  can  be  achieved. 

Finally,  three  underlying  factors  were  identified  by  use  of  factor  analysis.  The  first 
factor,  functionality,  relates  to  the  stmcture’s  ability  to  get  the  job  done  in  an  efficient 
manner.  The  second  factor,  repair,  reflects  the  maintenance  and  down  time  which  is 
inherent  in  the  structure.  The  third  factor,  variance,  refers  to  the  inherent  variability  of  the 
output  variables  measured  for  each  structure.  Some  success  was  also  achieved  in 
discrimination  between  large  versus  small  structures  and  IFR  versus  DFR  structures  by 
using  a  rank  order  of  the  repair  factor  (factor  2)  scores  and  variance  factor  (factor  3) 
scores  respectively. 

Throughout  the  discrimination  analysis,  consistency  in  the  results  was  observed  for 
each  of  the  various  methods  used:  similar  classification  accuracy  and  similar  best 
discriminant  variable  selections.  This  consistency  was  further  highlighted  when 
component/factor  score  rankings  were  used  as  a  discriminant.  For  example,  based  on  the 
DA  observations  one  would  expect  the  factor  which  represents  maintenance/repair  (factor 
2)  to  be  the  best  in  large  versus  small  discrimination.  This  in  fact  was  the  case,  with  the 
factor  2  scores  being  the  best  large/small  discriminant  among  all  factor  scores.  The  same 
proved  tme  for  factor  3  (variability)  and  IFR  versus  DFR  discrimination.  This  consistency 
in  results  provided  increased  confidence  in  the  conclusions. 
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methods  (pertaining  to  model  availability  estimation  error)  compared  to  using  automated  software  fitting  tools  (or 
‘wizards’).  The  results  of  this  stucfy  also  indicate  that  there  is  no  apparent  benefit  in  concentrating  on  ‘important’ 
components  for  the  highest  fidelity  distributional  fittings. 
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